In September 1989 Milton Friedman, the man whose views on economics influenced the policies of almost every government on the planet, wrote to Bill Bennett, "drug tsar" to the first President Bush. As Bennett prepared for a new phase in the "war on drugs", launched by President Nixon 18 years earlier – more police, harsher penalties, more jails, more military action overseas – Friedman wrote that "the very measures you favour are a major source of the evils you deplore". He pointed out how illegality made the drugs industry more, not less, lucrative, how crime had flourished during alcohol prohibition in the 1930s and would flourish more under Bennett's plans, and how "crack" might never have been invented had it not been for the drugs war.

Friedman was a firm supporter of decriminalising drugs, and regulating them as alcohol and tobacco are regulated. But however much governments listened to him on economics, they always ignored him on drugs. Many politicians of left and right have accepted the arguments for legalising drugs – but only before or after being in office. The signatories to a report launched in New York on Thursday, declaring that "the global war on drugs has failed" and that "the criminalisation, marginalisation and stigmatisation" of drug users should end, could hardly be more impressive.

They include former presidents of Brazil, Switzerland and Colombia, a former secretary general of the UN and a former US secretary of state. But the only current office holder is Greece's George Papandreou, who has other things on his mind just now. Other current leaders may be thought sympathetic. David Cameron said that the "war on drugs … has been tried and we all know it does not work". Barack Obama called the drugs war "an utter failure". But they said those things in 2002 and 2004 respectively, long before they got close to political power.

The arguments for legalisation are overwhelming. They do not rest on approval of drugs, or ignorance of their harms, or any wish to see their consumption increase. They are based on the argument that regulation would be less harmful to drug users, less damaging to society and less expensive to taxpayers than outright prohibition. Nobody disputes the dangers of drugs, only the best ways of controlling them.

All drugs become more dangerous when banned. First, because consumers have no protection from adulteration and often have no idea of the strength and quality of what they are buying. And second, because vendors favour more concentrated forms which are less bulky and easier to transport and hide.

Opium, smoked through a pipe, generates, as poets recorded, a drowsy numbness. Converted into pure heroin, a less bulky and more concentrated version, it does far greater harm, and is more addictive. Mixed with drain cleaner or sand – as much illegal heroin is – and injected into the veins with an unsterilised needle, it becomes lethal. During alcohol prohibition in America, consumption of beer fell 70% while consumption of wine and spirits soared. Alcohol was frequently mixed with methylated spirits, which explains the blind blues singers of that era.

Illegal drugs are also dangerous to those who never touch them. Because of the risks, suppliers charge premium prices, though, as in any retail business, new customers get bargain introductory offers. A drug habit is expensive and addicts turn to crime to finance it. Many become suppliers and join gangs which, because they operate in an unregulated market, protect market share and enforce contracts through violence. Estimates suggest over half of UK property crime is to fund drug misuse, and some judges reckon two-thirds of those in prison wouldn't be there if drugs were legal.

The war on drugs, then, is an expensive failure, an extended charge of the Light Brigade. At the time of the Misuse of Drugs Act 1971, the UK had perhaps 10,000 problematic drug users. Now there are at least 300,000. UN figures, quoted in Thursday's report, suggest that in the past decade annual global consumption of opiates is up by 34%, cocaine by 27%, and cannabis by 8.5%. According to the lobby group Transform Drugs Policy, legalisation of cocaine and heroin alone would deliver a net annual saving of £4.6bn (excluding any revenue from taxing these drugs as we tax alcohol and tobacco), even if their use were to double. Portugal, 10 years after it became the first European country to decriminalise the use and possession of all illicit drugs, has experienced only a slight increase in drug consumption, and a decline in heroin.

The arguments over drugs are done and dusted. Any independent body that looks at the evidence comes to similar conclusions. So why do political leaders refuse to countenance more than minor tinkering with the law, such as yo-yoing cannabis between classes B and C? One answer is that as Steve Rolles, senior policy analyst at Transform Drugs Policy, puts it, drugs have been presented as an existential threat and the war against them almost as a religious crusade. In the popular mind, drug users have always been demonised as what sociologists call "the other": Chinese gangsters, Caribbean immigrants, 60s hippies or other threats to the social order. Anyone who proposes ending the war risks being characterised by opponents, particularly in the downmarket media, as weak and cowardly, lacking the Churchillian spirit of "no surrender". History does not look kindly on those who lose wars.

But it goes, I think, even deeper than that. Control of drugs is deeply embedded in the DNA of modern government. The criminalisation of drug use, in the west at least, is almost entirely a 20th-century development. Laudanum, a tincture of opium, was in common use in Victorian England and Coca-Cola, invented in 1886, contained cocaine until 1903. No US state banned cannabis until 1915 and it remained legal in England until the 1920s, as did heroin and cocaine. The rise of conscript armies and Fordist mass production prompted the change, briefly affecting alcohol – the US took the first steps towards prohibition during the first world war – along with other drugs. Nobody wanted a drowsy numbness to overcome men marching into battle or clocking onto the production line. Significantly, Asian countries, which still earn their living from traditional manufacturing, now have some of the harshest anti-drug laws.

For most of the world, though, the time has come for political leaders to screw up their courage and rethink their policies. It surely cannot be beyond their spin doctors to present a switch to regulation not as a surrender but as a new phase in the drugs war. It is hard to think of anything that would do more to relieve death, destruction and human misery.