One of the tricks played by time is to turn the incredible into the inevitable. World leaders will gather in Germany tomorrow to deliver sonorous speeches celebrating the 20th anniversary of the fall of the Berlin Wall, that pivotal moment at the end of the Cold War when the starkest symbol of the Iron Curtain was physically torn down.

The implosion of Moscow's empire can now seem preordained. Soviet communism was so economically disastrous and politically atrophied that its totalitarianism was always destined for defeat by the prosperous liberal democracies led by the United States. For all the regiments of tanks and ballistic missiles that were paraded in Red Square, the Kremlin and the grisly henchmen who held central and eastern Europe in chains could never win. Such is the conventional wisdom today.

Yet the triumph of the west did not seem anything like so inevitable during the four decades of nuclear-tipped competition. The Soviet Union beat America into space in the late Fifties and early Sixties: first with Sputnik and then with a man called Yuri. "Whether you like it or not, history is on our side," bragged Nikita Krushchev. "We will bury you." Many western leaders feared that prediction would prove accurate. Harold Macmillan, Britain's Conservative prime minister, privately lamented to colleagues that communism's apparent ability to mobilise and direct its citizenry in a way free societies never could was likely to end in Soviet economic and military supremacy.

I am of the Cold War generation: not baby boomers, but atomic babies. I was born in the year of the Cuban Missile Crisis. The images, the tunes, the fashions, the badges, the rhetoric, the taste, the smell, the life of my childhood, teens and twenties is the Vietnam war, CND, Nixon in China, cruise missiles, Greenham Common, John le Carré, the Siberian gulags, Solzhenitsyn, Sakharov, the Russian invasion of Afghanistan, Solidarity, SALT talks, START talks, the four-minute warning, "duck and cover" and the hilarious "protect and survive" leaflets with their handy tips about how to use mattresses and cardboard boxes to build a refuge from radiation. My daughters are now taught the Cold War as history. For us, it was an ever-present. Always hovering at the dark edges of the mind, and thrust to the front of it at moments of tension between the superpowers, was the knowledge that a miscalculation in the White House or the Kremlin could climax with planetary annihilation.

One of the paradoxes of the Cold War is that it was the first global ideological competition which had the capacity to obliterate western civilisation and yet it was also the first struggle for domination to end peacefully. All previous confrontations between two highly armed and aggressively paranoid powers had ultimately come to a head with their armies clashing on battlefields. These two fought each other by proxy, but never directly. The Cold War is unique in ending because one side simply gave up. It did not seem at all inevitable until it happened that the Soviet leadership, under the enlightened Mikhail Gorbachev, would not react violently to the break up of its empire.

The shattering of the Iron Curtain was an epochal event which not only stood down the nuclear confrontation that threatened to destroy the world, but also liberated many millions of people. The hope generated by the popular and velvet revolutions in Europe helped to fuel an extraordinary burst of international optimism in the years immediately afterwards. Nelson Mandela was released from apartheid's prison to lead another remarkable transition in southern Africa. The fragmentation of the Soviet bloc reheated some conflicts previously frozen by the Cold War, leading to vicious ethnic warfare in the former Yugoslavia. It was an unalloyed good in many other regions of the world. Both the US and the USSR had prosecuted their rivalry by sustaining diabolical client regimes and brutal insurgencies in Africa, Asia and South America. The fall of the Wall was followed by the growth of democracy and respect for human rights in many lands previously barren of both.

In the west, the event initially proved to be better for parties of the left than of the right. The Cold War had split the left over nuclear weapons and the Atlantic alliance. The right would also taint western socialists and social democrats by association with the desiccated dictators in Moscow. This was made easier by the small minority of Labour MPs who actually were apologists for or fellow travellers with Moscow. I recall Margaret Thatcher shrieking across the Commons at Neil Kinnock: "He's a socialist – a crypto-communist!" This was outrageously untrue, but a card played with some effect by the right. A year after the Wall was pulled down, so was the Iron Lady. Not long after that, the White House was in the hands of Bill Clinton and then Number 10 was captured by another centrist progressive, Tony Blair. The price these new left politicians paid for power was to genuflect to the post-Wall consensus that unrestrained capitalism had become the all-conquering ideology of the world.

That victory was most hubristically celebrated in Francis Fukuyama's look-at-me book, The End of History, in which the American political economist proclaimed that we had reached "the end point of mankind's ideological evolution and the universalisation of western liberal democracy as the final form of human government". He was right to celebrate the defeat of communist tyranny. Liberal democracy is the worst form of government – except for all the other ones. But the 20 years since have proved him wrong to suggest that we could stamp a full stop on history. More of the world lives in free societies today than was the case two decades ago. Much too much of the world still doesn't: from Burma to Zimbabwe, from Afghanistan via Iran to Saudi Arabia. Religious fundamentalism and violent nationalism again challenge the values of liberal democracy. The world's most populous country remains a one-party state. China drew a quite different lesson from the defeat of the Soviet Union and adapted its communism into a form of nationalistic, authoritarian capitalism. Depressingly, China has also moved into the business of sustaining dictatorships, especially in Africa, to get its hands on resources.

Capitalism rampant has been spread across the planet by globalisation. On the positive side, it is estimated that half a billion people have been lifted out of subsistence-level poverty into decent lives by the economic growth of the past two decades. In the negative column, Karl Marx has been proved at least partly right about the capacity of capitalism to sow the seeds of its own destruction. As a result of the recklessness of the financiers and the ineptness of the politicians who failed to control them, we are now in the most severe recession since before the Cold War.

Victory also proved to be a more mixed blessing to the United States than Washington anticipated when George Bush Senior proclaimed a "new world order" to be policed by the sole superpower. By the standards of hegemonic powers, the US was a relatively benign behemoth immediately after the fall of the Wall. An alliance of more than a hundred nations joined its first war against Saddam Hussein to throw him out of Kuwait. Then came 9/11 and the termination of post-Cold War optimism. Driven by a cocktail of fear and hubris, George Bush Junior launched the second war against Saddam. The Iraqi tyrant was easily toppled, but America was catastrophically unprepared for handling the aftermath and is still thrashing around to find a coherent strategy in Afghanistan.

America's status is now challenged by other powers who have risen in the wake of the Cold War. The ascending nations – such as Brazil and India – demand a better seat at the table of international governance. The broader and more global G20 is rightly supplanting the old, narrow, western rich clique of the G8. Yet most international institutions – notably the composition of the UN Security Council – remain Cold War relics in desperate need of modernisation.

The spoils of victory did not turn out to belong to the United States so much as to Europe. The European Union has expanded eastwards to entrench freedom and spread prosperity across its continent. The EU now embraces 27 states, including virtually all of the former Warsaw Pact countries outside Russia. It has done so without having to point a single tank at anyone. This is a fantastic triumph of soft power and one to remember every time its affairs and Britain's relations with the EU are infantilised by squabbles that are trivial in comparison with that achievement.

Twenty years on, there are respects in which the planet has become a more dangerous place. America and Russia have reduced their nuclear arsenals and promise to slash them further. But other states are busy going ballistic. Mutually Assured Destruction helped to prevent the Cold War from ever going hot, but I would not be so confident of getting through another 50 years without a mushroom cloud when nukes are possessed by Pakistan, North Korea and Iran. The threat of global nuclear holocaust ended with the Cold War only for another existential menace to accelerate. Death by nuclear winter has been succeeded by the peril of the world being fried by global warming.

Twenty years on, the planet is richer, freer, more multipolar, less predictable and still pretty scary – but not quite as terrifying and definitely to be preferred to the world which was divided by that cruel Wall.