David Miliband claims he has no regrets – but being bolder would have taken
him to Downing Street, say Rosa Prince and Gordon Rayner.

From the moment Ed Miliband left the Labour conference stage on Tuesday, it was clear that his brother David's future was no longer in his own hands. By trashing the Iraq war that David had so slavishly supported, Ed was sending a blunt message to his sibling: you're not wanted.

The fury in David's eyes as he spat a rebuke to the clapping (and pro-war) Harriet Harman betrayed the realisation that, for the second time in a week, he had been skewered on live television by his little brother.

On Saturday, after he was beaten to the Labour leadership by a whisker, he had managed to hide his devastation as he praised and congratulated Ed. But Tuesday's public humiliation proved too much. After leaving the conference hall in Manchester, he turned off his mobile phone and caught the first available train to London, knowing his career in frontline politics was over.

David Miliband is by no means the first politician to discover that being thought of as a leader-in-waiting can be a curse. Rarely, however, has a party's leading light shone so brightly before being snuffed out so quickly or so cruelly.

"I'm not dead. I'm still here," he joked, as he announced his decision to retreat to the back benches. It was a brave performance, but one that became utterly unconvincing when he insisted that he didn't regret his decision to remain loyal to Gordon Brown, though many believe the 45-year-old could easily have seized the keys to Downing Street from him.

For eight years, a seemingly unstoppable momentum had been propelling Miliband towards the leadership. Yet he himself had never managed to shake off nagging doubts that he was up to the job; doubts that surfaced repeatedly when he was presented with a series of chances to take it by force.

From the moment in 1994 when Tony Blair recruited him from the Institute for Public Policy Research to become his head of policy at 29 (though he "looked about 12", Blair noted in his autobiography), Miliband caught the eye of the architects of New Labour.

Alastair Campbell nicknamed him Brains, after the geeky Thunderbirds puppet, when he was promoted to become head of the Prime Minister's policy unit in 1997, and a series of impressively argued pamphlets on the direction of the party persuaded Blair to hand Miliband the safe Labour seat of South Shields for the 2001 election.

Barely a year later, he was appointed schools minister, and after adeptly riding out the storm over 30,000 wrongly graded A-level papers, senior party figures heretically began musing over the possibility that Miliband could one day thwart Brown to become "the next Prime Minister of Britain", as one of them described him in 2002.

In May 2007, a month before Blair's pre-arranged exit from Downing Street, Miliband wondered if his time had come, and sought Blair's advice on running against Brown for the Labour leadership. "He was hesitant and I felt fundamentally uncertain as to whether he wanted it," Blair wrote. "And that is not a job to be half-hearted about."

By then, Miliband had racked up experience in four Whitehall departments; following his successful stint as schools minister, he served as a Cabinet Office minister, before joining the Cabinet as Communities and Local Government minister in 2005 and then as Secretary of State for Environment, Food and Rural Affairs. He was not only seen as highly competent, but had also managed to rid himself of his nerdy persona to become a polished media performer – one of the essential qualities for any prospective leader in the era of 24-hour news.

Blair told Miliband that if he challenged Brown, "I think you might win, not obviously but very possibly." Blair believed that if Brown was put under pressure, his "ambiguities" and "gaps" in his policies would be flushed out, putting Miliband in a strong position. "But David was unconvinced," Blair wrote. The rising star prevaricated, then turned his back on the fight, settling for promotion to Foreign Secretary in Brown's first Cabinet.

There was another, more personal reason for his reticence. At the time, Miliband and his American-born wife Louise were in the process of adopting their second child, Jacob, in the US. Bruised by the media storm surrounding their first adoption of an American baby, and fearing an escalation if he was Prime Minister, Louise persuaded him – to her eventual regret – that the time was not right.

Miliband will have the rest of his life to wonder whether this was the moment the biggest job in politics slipped through his fingers for good, but it was by no means the only occasion on which he decided to sit on his hands, rather than clenching his fists for a fight he might have won.

In 2008, Miliband, increasingly frustrated by Brown's failing premiership, tested the water for a possible leadership challenge by writing an article for the Guardian in the wake of Labour's dismal loss of the previously safe seat of Glasgow East in a by-election. He used the opinion piece to set out his vision for the future of the party, but pointedly failed to mention the Prime Minister at all, prompting frenzied speculation about an imminent coup.

While publicly denying that he coveted the top job, he made clear his intentions in a text to Lord Mandelson. "Large mountain ahead. Orienteering/climbing/planning skills much needed," he wrote. Miliband's secret plans were so advanced that he began musing on who might serve in his cabinet – telling friends he would love to make Alan Milburn his chancellor. And this time, his wife was behind the bid.

Again, however, Miliband dithered, blaming his Cabinet colleagues for failing to wield the knife by openly backing a contest. "There's a lot of 'After you, Claude' going on," he whinged to Mandelson. Rather than make him more determined to develop a killer instinct, the putsch that never was only increased his nervousness when a third chance to supplant Brown fell into his lap a year later.

Already undermined by the MPs' expenses scandal, Brown was there for the taking in June 2009 when a disastrous set of local election results led to the resignation of a dozen ministers, including Miliband's great friend James Purnell, who quit as Work and Pensions Secretary with a savage appraisal of Brown's ability to lead.

For 24 hours, the Labour Party held its breath as it waited for Brown to resign or Miliband to administer the fatal blow, but yet again Miliband was paralysed by indecision.

Publicly, he pledged loyalty, but in private he repeatedly asked friends if he had done the right thing, as if hoping someone would make the decision for him. Again, the moment had passed. By the time Geoff Hoon and Patricia Hewitt made a half-hearted attempt to topple Brown six months later, by calling for a secret ballot of Labour MPs, it came as no surprise that Miliband decided not to follow them over the top.

In part, Miliband's decisions had been born of a desire to run in a legitimate leadership contest rather than a bloody and potentially divisive coup, but they were also informed by his utter confidence that he would win hands down when it came to choosing a replacement for a defeated Gordon Brown.

There was one thing, however, he hadn't factored into his calculations; something he only discovered when he phoned a former Downing Street staffer to ask him to work on his campaign. "Sorry," the aide said. "I'm already working for Ed." It was the first confirmation of what until then seemed a remote prospect: that he would have to fight his own brother for the leadership.

Ed had spotted his brother's fatal weakness and sensed that his own moment had come. He also knew that he would never get another chance to lead the party; if David failed, Labour would surely not choose another Miliband to replace him.

Although the Milibands spoke publicly of their brotherly love, and David committed himself to a clean, open campaign, Ed had other ideas, showing that he possessed the killer instinct his elder brother clearly lacked.

David had been unprepared for the war of attrition his brother would unleash, particularly when Ed repeatedly questioned the wisdom of the war in Iraq. By the time the majority of the trade unions came out for Ed in mid-summer, David's team was beginning to get rattled. On the wall of his campaign office, they hung a series of portraits of Labour leaders – the last featuring Ed Miliband underneath the word "focus" – a prescient warning that the younger brother was not to be underestimated. But even as polls last week suggested Ed might shade it, David himself refused to believe that he could lose. In truth, however, his chance had come and gone years before.

Does he now wish he had ousted Gordon Brown while he had the chance?

"I don't regret that," he said yesterday. "I made the right decisions for the party and for the country."