Even idiots get it right sometimes. There's a chap that sits near me at White Hart Lane who causes his neighbours some form of psychic damage at almost every match. The second anything goes wrong, even in a game we are winning, he knows what the problem is and who is to blame. At the top of his voice (and it is a very loud voice) he overrides all the applause with: 'Get off the pitch, X, you waste of space!' I swear the players can hear him.

The identity of X may change, but there always is an X. And my man, who has no obvious tactical nous, not only sees what the manager does not, but feels entitled to offer his advice in the most intemperate tones.

At the beginning of this season, the new manager, like his predecessor, played one player in match after match. Motormouth objected as usual and the rest of us endured his tirades. Then the new manager departed and the new-new manager took over. X was immediately dropped and the improvement was marked; the shouter was vindicated.

So it is with politics. Since Thursday's election, every time I think about it, the result has looked worse to me. I can find nearly no comfort in the electorate's choices whatsoever, despite the size of the Labour majority. The share of the vote makes it impossible, except in the most legalistic sense, for the government credibly to claim a mandate. The campaign was fascinating, but was very damaging to Labour. The confected controversy over the Attorney General's leaked advice, for example, took up more time than discussion of education.

Now the Prime Minister threatens us with listening. Nothing makes me more worried than the demand that politicians should listen. For a start, some people talk much louder than others and are far easier to hear. Take top-up fees, an issue on which the Lib Dems probably gained tens of thousands of Labour votes. Nowhere during the campaign did I hear or see the question of support for poorer students raised with candidates or in the media. I would think that most people simply have no idea that these students will not have to pay fees and will receive, for the first time for years, a substantial maintenance grant.

The issue didn't come up because the parents of such poor students don't work in journalism and they won't write to the papers or go on marches. The redistributive nature of top-up fees has been successfully obscured by middle-class self-interest. In the same way, the Iraqis who want British troops to remain while they build their country are not heard with the same Lib Dems arguing for withdrawal, no matter what the situation is.

How do we 'listen' to those Iraqis, Darfuris, Africans? Yesterday, an editorial in the Guardian welcomed Mr Blair's 'listening' speech and praised 'its stress on the largely "domestic priorities that people want"'. This is so short-sighted.

At the same time as listening, the government is enjoined to be bolder. Which generally means being bolder about doing stuff that we want that offends others, and listening means being more timid when it comes to doing stuff that others want and that would offend us.

One of the things I hated about the election was its crossness. And this was by no means just about Iraq. Why isn't this solved? Why hasn't that been achieved? Ask not ever what you can do for your country, ask only what it has failed in the last five minutes to do for you. The new hospital is five miles away; the parking wardens are Nazis; my cousin's boyfriend's little brother had his mobile phone snatched at knifepoint. And it is all the responsibility of government, as though - on election - it had written out a gigantic money-back guarantee on every aspect of everyone's lives.

And government connives with this notion. So, in the listening speech, the PM identified the public's concern over the decline in civility. As if the ruderies were the fault of a different public, other people's kids! What will the government do, then? Institute national civility day? Are the censors going to move in on the films, magazines, TV programmes and other both popular culture (even high culture) that have contributed to the decline of mutual respect? Or are we just talking super-Asbos?

What we need is government that addresses the strategic and the difficult, not government that 'listens'. Labour has, when it has been at its best, done just that. It has understood that it is useless simply putting more money into unreformed and occasionally self-serving public services. 'Listening', by contrast, could easily mean failing to deal with entrenched vested interests. It could mean putting off the hard decisions on pensions, such as compulsory saving and later retirement.

Tony Blair has, by and large, understood this. I believe that he has been a very good Prime Minister, a supremely talented politician and, on the international stage, an enormously courageous one. I look on with despair at a public culture that vilifies him for his actions following 11 September, but that simultaneously rewards Saddam's principal apologist in Britain. It won't be easy to explain that in Basra or Kurdistan.

But it is obvious to me now that Tony Blair cannot be the vehicle to carry things forward. Just as in 1994, the right calculation was made that Blair was the person best placed to win support for change, now the self-same calculation says that he isn't. The combination of the assassins, of time, of blame, and his own inevitable errors mean that someone else must take on the job. At last the man with the empty head and the loud mouth has got it right.

Already some on Labour's backbenchers are giving notice of rebellion. A collection of tattered men o'war and patched sloops is firing directly over the decks of the old admiral's flagship and into the area of HMS Brown. Lynne Jones MP warns the Chancellor that the Lib Dems are 'tapping into core Labour values' and that they should be quickly re-embraced (ironically on the same page, in the same paper, the Lib Dems' Vincent Cable writes about the need for his party quickly to untap from such values, for fear of losing out to the Conservatives).

The last thing Labour needs is a reversion to the sterile triangulation between the demands of the party and the demands of the voters. This can best be avoided by a leadership election in which Gordon Brown stands on the basis of a commitment to change and reform in public services, to education, to taking the necessary hard decisions on pensions and to an intensification of the internationalism of New Labour on Africa, on Europe, actively promoting environmentalism, democracy and social justice abroad. Winning on such a platform, Brown will be entitled, as Tony Blair will not, to demand support, not just from Jones, but even from supposedly lovable buffoons such as Bob Marshall-Andrews. If the rebels have the courage of their own convictions, they will stand a candidate against Brown, and be trounced.

For some the moment of change cannot come soon enough. Robin Cook, the physical embodiment of impatience, wants Blair to step down within a year and is handing out daggers below the Senate steps. But that approach is unstrategic. Labour has, whatever the figures, just won an election. It would seem disrespectful of the electorate for the party leader to stand down too soon after even such a qualified victory. And though there is a need for change, there isn't a need for panic.

If the PM were to give notice of his resignation as party leader in June 2006, with a party election (with mandatory one-person, one-vote balloting among the trades unions) completed at the conference that autumn, it would give Gordon Brown time to prepare his manifesto and his premiership.

Despite all that, within weeks, the metaphorical bloke in the seat near me will be shouting, 'Get off the pitch, Brown!' But this time, he'll be wrong.