(CNN)Normally, science advances by trial and error. When an experiment fails, researchers question assumptions, formulate new ideas and then design better studies. But the field of nutrition is having a hard time of this when it comes to the low-fat diet debacle.

As noted in the Journal of the American Medical Association, conventional wisdom used to be that cutting back on fat would make us lean and healthy. However, things have not quite worked out that way, and Americans are struggling to adjust to a new dietary reality.

David Ludwig

Fifty years ago, we consumed a high-fat diet, with whole milk, rich sauces and spreads, full-fat salad dressings, regular peanut butter and fatty meats. But this eating pattern came under persistent attack, as preliminary research suggested that the fat in our food would make us gain weight and clog our arteries. Notably, fat has more than twice the "energy density" (calories per gram) than carbohydrates like starch and sugar. In addition, short-term laboratory experiments suggested that fat may be inherently less filling, causing "passive overconsumption."

Early clinical trials provided some support for this line of thinking. Volunteers adopting a low-fat diet spontaneously lost a small amount of weight for a short while (although these initial studies typically lacked an appropriate control group). And simple observational comparisons reported that people eating more carbohydrates tended to weigh less than those eating more fat.

Based primarily on this evidence, many experts proposed that replacing fat with carbohydrate -- any carbohydrate -- would naturally help us eat less and control weight without consciously trying to cut back calories. And this included sugar.

Read More

Like fat, sugar is tasty, but like all carbohydrates, it has a much lower energy density. From this energy balance perspective, sugar came to be seen as a good way to displace fat and calories from the diet -- the so-called sugar-fat seesaw. "The evidence intriguingly suggests," one preeminent team wrote, "that it is specifically an increased intake of sugars . . . rather than of complex carbohydrates that tends to dilute fat energy." Another team wrote that, "fat promotes overconsumption while sucrose [table sugar] probably prevents it."

In a very real sense, it seemed we could have our (low-fat) cake and eat it, too.

Despite concerns for the lack of high-quality scientific evidence, the government and all the major professional nutrition associations had by the 1990s recommended that everyone beyond infancy eat a high-carbohydrate/low-fat diet. Americans were told to substitute all fats with a variety of carbohydrates, including six to 11 servings of grain products daily, as exemplified by the original Food Guide Pyramid.

To facilitate this change, the government's Healthy People 2000 goals officially called upon the food industry to increase "to at least 5000 brand items the availability of processed food products that are reduced in fat."

And so we substituted whole milk, peanut butter and regular salad dressing with sugary reduced-fat versions. We avoided nuts, avocado and dark chocolate, while loading up on a variety of low-fat treats and sweets. As a result, within one generation, the proportion of fat in our diet decreased from above 40% to near the government-recommended 30%. But rates of obesity and diabetes surged, and the decades long downward trend in heart disease seems poised for reversal.

According to the energy balance hypothesis of body weight control, substituting carbohydrate for fat should have reduced rates of obesity and diet-related chronic disease, but the opposite occurred.

Contrary to prior thinking, new research shows that many types of fat are highly filling and exceedingly nutritious. Some of the most calorie dense foods in existence -- nuts, fully fat dairy, olive oil, dark chocolate -- are associated with less weight gain than processed grains, potato products and other commonly-consumed high-carbohydrate foods.

Several recent systemicreviews show that high-fat diets produce greater weight loss than low-fat diets, when both groups in the trials are given equivalent support.

Most importantly, reducing fat intake did not lower rates of cardiovascular disease in two major clinical trials, Look Ahead and Women's Health Initiative, whereas increasing fat intake in the Predimed Mediterranean diet study did. Consistent with these findings, a study this year found that people consuming a high-fat diet had 16% lower rates of premature death than those consuming a low-fat diet (although the type of fats played a significant role in determining risk).

Responding to new evidence, the 2015 USDA Dietary Guidelines lifted the limit on dietary fat, unofficially ending the low-fat diet era. But you'd never know it, because a full accounting of this failed experiment has not been made. In the absence of this corrective process, public health harms persist, with the low-fat diet remaining deeply embedded in public consciousness and food policy. In fact, according to a recent Gallup survey, a majority of Americans still actively avoid eating fat.

It's time to acknowledge past mistakes and examine why a focus on calorie balance backfired. One explanation is that the body fights back against calorie reduction, with rising hunger and slowing metabolism, making it increasingly difficult for most people to maintain weight loss on a conventional low-fat, low-calorie diet. But colleagues and I have argued that all calories are not equal. By reducing consumption of processed carbohydrates, insulin levels fall, unlocking calories stored in fat and helping promote long-term weight loss (the carbohydrate-insulin hypothesis).

If this alternative view is right, it would mean that calorie restriction is useless over the long-term, and that weight loss treatment should focus on the type, not amount of calories consumed -- the opposite of the conventional energy balance recommendation.

But this potentially exciting scientific debate has been mired in revisionist history, detracting from a clear contest between these two contrasting hypotheses. Disregarding extensive evidence to the contrary, some claim that no low-fat advocate ever recommended fat-free junk food -- it was the food industry's fault for marketing these products and the public fault for buying them. But if the actual intention of prior dietary recommendation were to increase vegetables, fruits and whole (instead of processed) grains, there would have been no need to limit fat in the first place.

Others call for defunding low-carbohydrate diet research because their benefits for body weight don't seem large, but this is exactly the wrong medicine. In fact, studies of alternative diets have received miniscule governmental funding compared to research into the low-fat diet. For this reason, most studies suffer from important limitations such as use of ineffective methods to actually change diets.

This debate has also been unnecessarily complicated by ethical and environmental concerns about eating meat. Though these concerns are important, they rest on the false premise that all high-fat diets are inherently high in animal products. In practice, one can eat a low-fat diet with lots of lean red meat, poultry, reduced fat cheese and egg whites; or a high-fat diet with olive oil, nuts and other plant-derived fats.

The science of nutrition is complex. But we know that the low-fat diet of the last 40 years didn't work. In view of the human and economic toll of diet-related disease, this failure warrants a rigorous examination, efforts to mitigate existing harms and robust government funding to test new ideas.