So much for the decades in which fats and oils were public enemy number one on our dinner plates. There is more and more evidence that sugar – or more precisely, carbohydrate – is behind our increasing rates of obesity and heart disease. Even if the mechanisms by which this occurs are still not well defined, there are endless calls for reducing its quantities in the foods we eat. Most recently in the UK this led to the chancellor, George Osborne, announcing a tax on sugary soft drinks.

Had we ever come up with a proper substitute for sugar, of course, we wouldn’t need to have this debate. In our sweetness-addicted era, it is one of science’s greatest challenges. So why has it eluded us for so long, and are we any closer to a solution?

Replacing the sweetness of sugar in foods is actually relatively straightforward. The first synthetic sweetener, saccharine, was discovered accidentally by a young Russian chemist named Constantin Fahlberg in 1879 while studying coal-tar derivatives, when he unknowingly got it on his hands and licked his fingers. Saccharine became widely used around World War I, when natural sugar was in short supply. In the 1960s scientists discovered several more artificial sweeteners in similarly serendipitous ways, including aspartame and acesulfame K.

As well as these discoveries, there are naturally occurring sweeteners that we have actually known about for much longer (see table below). The Guarani peoples of modern-day Brazil and Paraguay have been using the leaves of the stevia plant as a sweetener for about 1,500 years. And the seeds of the West African katemfe fruit, which contain a sweet chemical called thaumatin, have been on our radar since the 19th century.

Sweet but sour

Yet while we have plenty of options for sweetness, there are several difficulties associated with using non-sugar sweeteners in foods. There have been various cancer scares over the years, which have affected stevia, saccharine and aspartame, among others. Some artificial sweeteners have also been linked to type 2 diabetes.

To compound this, governments class all non-sugar sweeteners as additives, which means they are assigned an E-number – even stevia and thaumatin. In an era where consumers have become increasingly wary of these numbers even when there aren’t specific health risks, manufacturers have been moving towards so-called “clean-label” products that are free of them. This puts these sweeteners at a disadvantage.

Aside from health and labelling, sugars have chemical functions in foods that make them difficult to replace. Sugar solutions freeze at a lower temperature than pure water, for instance. In products like ice cream, this is critical to maintaining a soft texture at freezer temperatures.

Sugars play an important role in giving products like bread, cakes and even wine their darker colour, through what chemists call non-enzymatic browning reactions. Artificial sweeteners are not good at reproducing either of these.

Then there is aftertaste. This arises from the mechanism by which sweetness is detected in the taste buds. One problem is that the structural features of any sweet molecule that allow them to bind to the sweetness receptors on the tongue are similar to the ones that bind to our bitterness receptors. This is why some sweeteners leave a bitter aftertaste, which is of course undesirable to some consumers.

But looking at the previous table again, for sweeteners that don’t have a bitter aftertaste there is another issue. Artificial sweeteners bind more strongly to the sweetness receptors and have a different and longer-lasting taste profile to sugar, and so are perceived as tasting different by consumers.

All in all, although non-sugar sweeteners are a multi-billion-pound industry, these drawbacks help to explain why they are nowhere near eclipsing sugar. In 2014 sugar (sucrose) accounted for 78% of all sweetener sales. Artificial sweeteners made up 8%, with acesulfame k the market leader. Natural alternatives like stevia, which was banned in the US and EU until fairly recently, made up 1%. (The rest of the market comprises everything from glucose to syrups).

Where sweeteners go from here

The cancer evidence against non-sugar sweeteners has turned out to be thinner than feared. Cancer Research UK and the US National Cancer Institute both say there is no increased risk regarding artificial sweeteners. Stevia’s years in the wilderness were the result of an anonymous complaint about the cancer risks to the US authorities commonly thought to have come from artificial-sweetener producers, but it has since been rehabilitated. As for type 2 diabetes, the evidence linking it to artificial sweeteners is inconclusive and we need more research – so far it has all been done on animals.

On the physical issues, food scientists have had to think creatively. When it comes to texture, for instance, manufacturers add protein texturisers instead – soy, for example. Or you can turn to other substances that have a similar effect as sugar on the freezing properties of water – the sugar alcohol erythritol is one option.

Manufacturers seek to overcome the aftertaste issue by mixing sweeteners. We perceive the aftertaste of different sweeteners over differing timescales, so one sweetener can be used to mask the aftertaste of a second. It is common to use stevia in combination with acesulfame K, for instance.

Another increasingly common ploy is to mix sugar and other sweeteners together. This helps explain why the use of non-sugar sweeteners in new product launches rose from 3.5% in 2009 to 5.5% in 2012. It also explains why stevia is rocketing. Food analysts Mintel and Leatherhead forecast it will have become the most widely used non-sugar sweetener by as early as next year.

In the absence of a Holy Grail for sugar replacement, this could be as good as it gets any time soon. No wonder the authorities are beginning to intervene to save us from our sweet tooth instead.