In the mid-1950s, with the cold war growing chillier, paranoia seeped through the corridors of the Rand Corporation, the fabled military thinktank in California. After all, to the hotshot young analysts paid to devise America's strategy in the nuclear standoff with Moscow, paranoia seemed to make perfect sense. If you assumed that you couldn't trust your enemy - and you assumed that your enemy felt the same about you - then whatever noises you made about disarmament, you'd always stockpile weapons, because you'd assume your enemy was doing the same. Nobody would dare attack, and an edgy stability would result. Act with trust and co-operation, on the other hand, and you risked a situation where both sides would claim to be willing to disarm, but then only you actually did so, spelling instability, then doom.

This was what the thinktank's logicians called the "prisoner's dilemma", and the more ambitious among them - inspired by John Nash, the mathematical genius and Rand Corporation scholar portrayed by Russell Crowe in the film A Beautiful Mind - had high hopes for their newborn theory. Could it be, they wondered, that stability in everyday human relations was achieved by the same kind of self-centred suspicion and distrust? To test their ideas, they recruited the nearest everyday humans they could find: the Rand Corporation's secretaries. In experiments, they posed various dilemmas for pairs of secretaries, in which they could co-operate or betray each other. (A typical question involved the purchase of a Buick; one imagines women in knee-length dresses, gamely tolerating questions from clipboard-wielding men in horn-rimmed glasses and short-sleeved shirts.) The theory predicted they'd choose betrayal, because they couldn't trust the other one not to. Every single time, however, they chose to co-operate.

Perhaps if the analysts had paid more attention to their secretaries, the history of the past half-century would have proved very different. Instead, according to a new documentary series beginning on BBC2 next weekend, the paranoid theories hatched during the cold war would come to inspire a peculiar, cold-hearted idea of personal freedom - one that helps explain everything from the rise of Prozac and Viagra to Labour's obsession with healthcare targets, from the military crusades of George Bush and the rise of the Iraqi insurgency to the rampant diagnosis of attention deficit disorder in children.

This is an audacious hypothesis, even by the standards of the documentary-maker involved, Adam Curtis, whose 2004 series The Power Of Nightmares asserted that al-Qaida, as an organised entity, was essentially an invention of the west. The new series, The Trap: What Happened to Our Dream of Freedom, argues that we have unwittingly subscribed to a bleak ideal of liberty that has, ironically, "become our cage", reducing our true freedom and fuelling a dramatic rise in inequality.

Critics will probably accuse Curtis, as they did after The Power of Nightmares, of being paranoid himself - of seeing in government policies a sinister plot to control the populace by tricking them. "But I've never believed that anyone's bad," the 51-year-old Curtis insists, bouncing restlessly around the Soho office where he's editing the series. "People do bad things because they're forced into circumstances. Journalists always want to find a smoking gun - people sitting in rooms saying 'let's bomb Iraq'." As for what happened to the concept of freedom, he says: "I don't think there are baddies in this. I think our leaders, and us, in the belief that they were trying to find freedom, have gone down a road that's led us into a trap, towards a world without meaning or purpose. We're complicit."

The cold war way of thinking about human nature, mirrored by the work of the economist Friedrich von Hayek, inspired the nascent Thatcherites. They were convinced that civil servants and public-sector workers, while claiming to serve the greater good, were really just self-centred and out for their own gain. As in the nuclear standoff, it was best to be honest about the fact that everyone involved was cold and calculating; the dangerous people were the ones who claimed to serve some higher ideal. "We're safer if we have politicians who are a bit self-interested and greedy," says James Buchanan, the grandfather of this approach, "than if we have these zealots ... who think they know best for the rest of us." Hence the culture of public-sector targets, pioneered by Margaret Thatcher and massively expanded by Tony Blair: give people the right incentives, the theory went, and in pursuit of their own interests they'll end up helping everyone.

In a typical bit of conceptual long-jumping, The Trap leaps from politics to the radical Scottish psychiatrist RD Laing, who saw normal families as hotbeds of strategy and scheming, with husbands and wives manipulating each other as if they, too, were just like the White House and the Kremlin. Psychiatry abetted this nightmare, defining people as mad if they rebelled against the system.

In one famous proof that madness was defined by a patrician establishment, an American follower of Laing, David Rosenhan, arranged for eight healthy researchers, himself included, to check themselves in to mental hospitals. They claimed they could hear a voice in their heads saying "thud". All were diagnosed as ill; it took Rosenhan two months to get himself discharged. One hospital chief, defending the profession, urged Rosenhan to send more impostors and promised to detect them. He agreed, and soon the hospital was boasting the discovery of 41 fake patients. Rosenhan hadn't sent any.

But in trying to overthrow the old definitions of madness, Curtis demonstrates, many psychologists gave up looking inside people's brains at all. Instead, they devised a system of checklists based entirely on symptoms - the tome known as the Diagnostic and Statistical Manual. If you could tick enough boxes, you had the disorder: depression, obsessive-compulsive disorder, ADD, or the rest. The new way of thinking allowed people to diagnose themselves, and thus chastise themselves for experiencing ordinary human emotions, just because they diverged from an ideal of what was normal. They were free of a psychiatric establishment telling them how they should be - but they weren't really free at all. They had become enslaved by paranoid self-monitoring.

In earlier times, psychiatrist Paul McHugh tells Curtis, "people didn't want to see themselves as in some way psychiatrically injured. But now, they tell me that they have an ideal in their mind about what the normal person is. [They say]: 'I don't fit that model. I want you to polish me down to fit.'..."

Curtis argues that while the radical psychiatrists inadvertently ended up enforcing a single ideal of the normal human, so too did target-obsessed Thatcherites and Blairites begin to turn people into the calculating machines they'd wrongly assumed them to be. The precise results, however, proved unexpected. The theory of the "invisible hand" of the free market, first crystallised by Adam Smith, whereby the pursuit of self-interest results in order, may apply in the world of business - but not necessarily when artificially imposed elsewhere. Set a target for the reduction in patients waiting on hospital trollies, and NHS managers are liable to respond - as some notoriously did - by removing the wheels and reclassifying them as beds.

"I realise what I said at some times may have over-emphasised rationality," an elderly John Nash tells Curtis in an extraordinary interview, after emerging from years of battling schizophrenia. "Human beings are much more complicated than the human being as a businessman." In fact, the documentary notes sardonically, experiments show that only two kinds of people behave like perfect little economists in every arena of life: economists themselves, and psychopaths.

The Trap's argument won't convince everyone. The link between all these ideas and the way "freedom" was used as a justification for the wars in Afghanistan and Iraq isn't so clear. And the truth about public service workers may be that they are neither knights nor knaves, but a bit of both, argues Professor Julian Le Grand, one of the architects of Blair's health reforms. "The real trick is that you need a system of incentives that encourages the knight and the knave in each of us, and gets them working in the same direction," he says. But he agrees with the film's argument that if you keep treating people as if they were selfish and calculating, that's how they'll eventually become. "We ... come to believe," as Curtis puts it, "that we really are the strange, isolated beings that the cold war scientists had invented to make their models work."

Curtis enjoys an extraordinary latitude in his filmmaking: in a post-Hutton BBC, it was extraordinary that he was allowed to make The Power of Nightmares at all. (It put noses out of joint at the corporation, which commissioned another documentary, The New Al-Qaida, essentially a riposte.) "I love the BBC," Curtis says. "It's so complex that no-one can control it. My job is to swim in the chaos and use it to my advantage." The BBC had its own experience of internal-market culture, with similar effects to the trolleys-and-beds fiasco detailed in the film. "Actually, though, the Birtian revolution was fine," Curtis says. "It just meant more chaos."

The Power of Nightmares drew fire from some right-wingers, who accused Curtis of imagining conspiracies and objected to the parallel he drew between the evolution of neoconservatism and that of radical Islamism. In the US, the conservative National Review accused him of fuelling "Chomskyite visions of 'Amerika' as the fount of all evil". The criticism misses the mark. The al-Qaida film shows that Curtis believes, above all, in the political force of ideas; many "anti-imperialist" critics of western foreign policy see nothing at work but blind economic forces. Where he differs from the neocons is in pointing out that the consequences of their ideas weren't the ones they predicted.

"That sort of pissed me off," Curtis says of the conspiracy theory charges. "There's an affectionate tone in that series. I'm kind of taking the piss out of conspiracy theories." His trademark filmmaking style, in fact, undermines any too sinister interpretation of his subject-matter. The story of the neocons is narrated against clips from black-and-white horror movies, with appropriate scary background music.

It might also surprise his critics on the right that Curtis doesn't buy the standard anti-Bush critique that the "war on terror" turned the imaginary threat into a real one. "There's very little evidence that there actually has been the rise of an organised al-Qaida network," he says. From the media's point of view, he argues, "all you have to do is call yourself Al-Qaida In Islington, and you're part of an organised network." The trick "is to step back. If there's one thing that links all I do, it's trying to make people pull back, look at their time. All the news journalists now are so obsessed by the idea of 24-hour news that they have no idea what it all means."

The Trap occasionally feels as if it is stepping a little too far back, wrapping the whole past half-century into a single argument. As Curtis readily concedes, the old civil service did need replacing; a world in which psychological disorders are over-diagnosed has also seen thousands of lives transformed by Prozac. It's not that our old ideas of how to run society were any good; it's that our new ideas didn't work out as planned.

The most perceptive comment on the situation comes, in Curtis's film, from a beleaguered bus conductor, in archive footage used as a counterpoint to the visionary talk of targets and markets and freedom. It could serve as a general diagnosis of the problem of how best to approach politics, psychology, culture - the lot. "Anybody that deals with the public, you can never win," he says, flatly. "You can never win when you deal with the public. Never."

Under the microscope

1920: Little AlbertIn an experiment that wouldn't make it past any university ethics committee today, researcher John Watson tried to show how early experiences affect the kinds of people we become.

The nine-month-old experimental subject, Albert B, was introduced to a white rat, and a rabbit; he showed no fear. Then he was presented with the white rat while Watson hit a metal bar with a hammer behind his back.

After this happened several times, the baby became afraid of the rat - and of the rabbit, and of other furry animals and objects. Albert's mother never gave her consent for the experiment, and the baby left the hospital before any attempt could be made to eliminate his new fears.

1960s: The YanomamiThe anthropologist Napoleon Chagnon made headlines with his studies of violence among the Yanomami people of Venezuela and Brazil.

What looked like orgies of aggression, he concluded, actually followed a strict logic: tribe members defended those to whom they were more closely genetically linked, protecting them from those who were genetically more distant.

Doubters had another theory: that the violence was influenced by the presence of the Western anthropologists. The fighting, they said, was between village members to whom Chagnon's team had given machetes, and visiting tribespeople, who wanted machetes too.

1961: The Milgram ExperimentsThese notorious studies on obedience to authority, by the Yale psychologist Stanley Milgram, revealed what seemed to be a horrific dark side to humanity: if told to do unconscionable things by someone who presents themselves as an authority, we will.

Milgram's subjects were told they were helping to study the effects of punishment on learning, and would have to administer electric shocks to another so-called "subject", in another room, who was actually an actor.

The subjects proved willing to administer what they believed were electric shocks to the actors, up to and including fatal levels, because the person running the experiment told them that they must.

Even though everyone understood they were part of a simulation, chaos resulted as the subjects adapted with alarming speed to their assigned personas. Guards became genuinely violent and sadistic; the prisoners rioted, and showed signs of trauma.

Alarmed researchers called off the experiment early - reportedly to the consternation of some of the guards, who had come to relish their roles.