If you peruse news sites like Grist regularly, you’ve read about some amazing new gizmo that promises to make farming sustainable forever more. And, a few years later, you may have remembered that story and wondered why that innovation failed to transform agriculture. The answer is that farmers just didn’t buy it.

“There’s all this new innovative technology, why aren’t farmers just going crazy trying this?” asked Suzy Friedman, director of agricultural sustainability for the Environmental Defense Fund. It’s a rhetorical question for Friedman. She has the answer, and she’s just helped launch an initiative that could provide the solution to this problem.

Let me explain why farmers often hesitate to embrace new things with a thought experiment.

I’ve got an awesome new gizmo to sell you. Plug it into your car and you’ll be able to drive farther on less gas, so you’ll save money while reducing your tailpipe pollution. But here’s the catch: This gizmo is expensive, and it’s a gamble. It could improve your wages, but, if it fails, you won’t just lose the money you spent to buy it, you could lose all your take-home pay, too. Don’t worry about that though, because, as I think I mentioned previously, it’s awesome! Want to buy one?

That’s what it’s like to be a farmer.

The problem with revolutionary ideas for farming is that many of them don’t live up to their promises — when something sounds too good to be true, it usually is. And if a new tool fails to work, and decreases a farm’s productivity just a little bit, it’s likely to turn profits into losses. Every time a farmer tries a gizmo that messes with their yields they are laying a year’s wages on the roulette table.

“Show me one case where you have been willing to spend significant money on something that will affect your income for the entire year, without any data that it will work,” Friedman said.

Providing solid evidence that something works is the key, Friedman said. The Environmental Defense Fund, along with several partners, is tackling this problem by creating a kind of Consumer Reports for farm sustainability tools called NutrientStar. The goal is to show what works and what’s just snake oil.

NutrientStar is focused on reducing fertilizer loss — it’s basically a clearing house for assessments of fertilizer gizmos. Making fertilizer creates a lot of greenhouse gases, putting too much of it on a field creates even more, and when washed off fields it pollutes waterways and leads to algae blooms. There’s been an explosion of new products claiming to help farmers deliver fertilizer to crops more efficiently to remedy these problems. Now, you can go to NutrientStar to see which of these products have been scrutinized. So far, it only has a couple of reviews (it’s brand new): There’s one on a “slow release” fertilizer (verdict: not enough info to tell if it’s any good), and another on software to fine-tune fertilizer application (verdict: hey, it works).

For that software, a Cornell-developed system called Adapt-N, NutrientStar has a summary of randomized field trials. Farmers have traditionally applied the same amount of fertilizer across every square foot of their field. A tool like Adapt-N tells them to put a little more on one square foot, and a little less on another. The field trials show that farmers using Adapt-N reduced their nitrogen use by 37 pounds per acre, increased yields by two bushels an acre, and increased profits by $30 per acre.

That’s the kind of information that could give activists a sense of what they could push food companies to achieve. Data like that might even give farmers the confidence to invest in some expensive gizmo, or revolutionary idea.