No ascent to the prime ministership was more carefully planned and plotted for so long than that of Gordon Brown in 2007. Few were more enthusiastically promoted by Labour activists and media supporters as a decisive change towards better political times after the divisions of the Iraq war. None has been so difficult or so troubled in practice.

Some uncertainties about the timing of the handover remained when Labour won its third successive mandate in May 2005, returning to Westminster with a sharply reduced 36% share of the vote but with a healthy majority of 65 seats. But party officials dubbed the transition Project Cake. Compared with what came after, Brown's takeover was a piece of one.

Brown became Labour leader on 24 June 2007. As he had always wanted, and had gone to great lengths to ensure, he was unopposed – the first man to succeed to the premiership without serious party rivals since Anthony Eden succeeded Winston Churchill in 1955.

Brown's first cabinet seemed to mark an inclusive approach of which some had feared Brown was incapable. The crucial appointment was the consensual Alistair Darling as chancellor rather the divisive Ed Balls. David Miliband, who might have run against Brown for the leadership but did not, was rewarded with the Foreign Office. A so-called "government of all the talents" was marked by ministerial appointments from industry, the services and the United Nations. There was much talk of a return to cabinet government after the Blair years. Beneath the radar, however, Brown also brought his own trusted and often highly partisan officials in with him.

At first things went well. Terrorist incidents in London and at Glasgow airport in his first week of office were seen off. A high-profile response to floods and an outbreak of foot-and-mouth disease reinforced the message that an apparently strong leader had taken over. Brown gave the impression of having arrived in office with a check-list of Blair's negatives and was proceeding to neutralise each one in turn: everything from Iraq to sofa government, the scrapping of a planned super-casino, tightening the law on cannabis.

In time, this approach would come to be criticised as reflexive micromanagement. At the time, though, it was seen as a tonic. The media were impressed. Brown was rewarded by a powerful boost in the polls.

In reality, however, things were much less certain than they appeared on the surface. In particular, it was not clear what, other than not being Tony Blair, Brown was really offering. As the Labour general secretary Peter Watt wrote bitterly later: "Everyone around him thought there was some big plan sitting in a bottom drawer somewhere, just ready to be pulled out when the moment came. In fact, there was nothing."

One of the many issues that Brown had never resolved in his own mind was whether he would call an early election after becoming leader or whether he would "go long" into 2009 or even 2010. Now, in early autumn 2007, with the polls moving Labour's way, election talk grew suddenly louder.

As Labour gathered in Bournemouth for Brown's first conference as leader, he put David Cameron on the back foot by embracing traditional Tory positions on crime and have-a-go heroes and announced his support for "British jobs for British workers". Labour led by 11 points. But a Tory fightback, centred on a pledge to slash inheritance tax, stopped his advance. Labour's poll rating dropped. Brown was forced to announce there would be no election after all, and gave a barely believable interview implying he had never had much to do with the idea.

It was a turning point for Brown's standing as a political leader. His reputation looked even worse as the full impact of the revolt on the 10p tax rate abolition kicked in that autumn and winter. The loss of government computer disks full of citizens' tax information added to the change of mood. So did a Labour party funding scandal. The misery was compounded by a cack-handed handling of the Lisbon EU treaty signing ceremony and the arrival in London of the Olympic torch from China.

Brown's mood collapsed from confidence to anger. Reports of his private fits of rage began to seep into the public domain. Darling delivered a sober budget in March 2008, signalling that times were getting harder. In May Labour lost a byelection to the Tories for the first time in 30 years, at Crewe and Nantwich. On the first anniversary of Brown's succession, Labour came fifth in the Henley byelection. By now Brown's reputation had come full circle.

The second half of 2008 was dominated by two contrasting but equally compelling dramas. On the one hand, there were recurrent surges of speculation about whether Brown's leadership would be challenged, probably by David Miliband, possibly by the increasingly well regarded health secretary (as he then was), Alan Johnson, perhaps even by Jack Straw. On the other hand, far more important for the whole country, there was the dizzying descent that autumn into the most serious financial crisis in living memory, marked initially by the collapse of the US banking group Lehman Brothers, and which culminated in October's taxpayer bailout of RBS and Lloyds TSB. In the event, the latter would help Brown to see off the former. Brown's handling of the financial crisis was sufficiently decisive for him to regain the aura of successful leadership.

But his political recovery was never as strong as he and his officials tried to pretend, and Brown's standing has never entirely recovered. For much of 2008 and 2009, Brown was caught between the twin pressures of Conservative resurgence and continuing Labour doubts about his leadership. One of his few undoubted successes came in October 2008, when Brown reshuffled his cabinet and brought back his long-time enemy Peter Mandelson as business secretary and, soon, as de facto deputy prime minister. Mandelson's return was a masterstroke, binding the Blairites into the Brown government afresh and strengthening the cabinet at the height of the financial crisis. But it was also a sign of weakness, an implicit acceptance that the man caricatured in Private Eye as the Great Leader was increasingly dependent on others.

Brown's poll numbers began to recover towards the end of 2008 in the so-called "Brown bounce", though they then eased. Brown tried to capitalise on his success in the spring, by hosting a meeting of the so-called G20 economic powers in London. But these efforts never quite achieved what was intended. Partly this was because of Brown's continuing gracelessness as a leader. His comparison of himself with Barack Obama as a leader embodying hope and change misfired, a YouTube video in which Brown smiled at strangely unpredictable moments was much mocked, and when he inadvertently told the Commons that he had saved the world, many believed that this was precisely what he thought.

Time and again, Brown seemed slow to see dangers coming, as on the treatment of the Gurkha veterans or as the MPs' expenses crisis developed during 2009. The contrast between the elevated, morally focused politician that Brown affected to be and the angry grudge-bearing partisan about whom enemies (and one or two friends) complained was humiliatingly highlighted by the revelation that Brown's longtime acolyte Damian McBride was planning crude email slurs against Cameron and his shadow cabinet. McBride was forced to quit.

By spring 2009 things were as bad for Brown as they had ever been. The war in Afghanistan became a focus for a particularly personal goading of Brown by the tabloid press over equipment failures and continuing deaths. The local and European elections were catastrophic for Labour. That week Brown had to reshuffle his government again, but was faced by an outright challenge from his work and pensions secretary, James Purnell, who resigned very publicly with a call for Brown to quit too. Brown had to call in Mandelson to ensure that Purnell's example was not followed, above all by David Miliband. Nothing revealed Brown's weakness more than his very public wish to replace Darling at the Treasury with Balls. Darling's refusal to take any other post in Brown's government stymied the move – and confirmed Darling as an increasingly substantial figure.

At the end of 2009 disaffected ex-ministers launched yet another unavailing challenge to Brown, which he saw off with ease. Yet as the election drew closer, Brown seemed to rediscover some confidence. No longer an electoral asset for Labour, he nevertheless commanded grudging respect in many quarters. Brown's failings as a leader were obvious and well-documented. But so were his strengths, his grinding energy, his puritanical high standards over money throughout the expenses crisis, and his unflagging seriousness.

He remains an indisputably substantial, even protean, political figure. If the latest uptick in the Guardian-ICM polls proves a good guide, he remains in with a shout of leading Labour into a strong position in a hung parliament. Perhaps he may even pull off the most improbable electoral win in Labour's history. If that happens, then Brown's problems will not be over. But the political agonies of the past three years will seem to have been all worthwhile.