In the end it was not Red Ken versus Blue Boris, it was simply Boris against Ken. In a city that routed the Tories in its assembly election, the personality factor loomed larger than the call of the old tribes in the mayoral race.

It was closer than some had predicted, but Londoners preferred the shambolic incumbent with a way of raising a smile to the old hand making his fourth straight run for this particular job, some 31 years after he had first run the capital through its long-abolished council. Ken Livingstone's undoubted past accomplishments – such as the congestion charge and paving the way for civil partnerships – were overshadowed by a talent for creating divisions. Last night he looked like a man whose great come-backs lay behind him. Boris Johnson's grip of pressing responsibilities, such as housing, is faltering at best, but he has proved less catastrophic than some had feared. Encouraged by a media that leaned his way, Londoners have decided to hang on to a champion whom they can trust not to swallow the party script and never to make the deadliest mistake in contemporary politics – which is to sound like a politician.

Amid a revolt against the old partisan order – a revolt evident in strong showings for Ukip and the Greens – David Cameron's promise of maverick mayors all round might have been expected to resonate, but his rallying cry of "a Boris for every town" fell flat. The coalition proposed directly elected mayors for England's dozen biggest cities outside the metropolis, but of the 10 which staged referendums on Thursday, nine flipped their thumbs down. There will be a knock-on effect on the move to elect police chiefs. That scheme, too, would surely have been rebuffed if it were on the ballot. As it was not, though the elections will go ahead in the autumn, without a simultaneous poll for city mayors, the turnout will be dire.

Across the north, resistance to appointing another well-paid politician killed the idea. In Birmingham, meanwhile, Yes campaigners focused too much on business support, as opposed to reaching out to the public. Britain's second city was one of several in which divisions in council party groupings played a part – with canvassers barred from mentioning the mayoralty for fear of opening factional divisions. Questions about the power balance between the mayor and the council were never convincingly dealt with and even those few voters paying attention were left without any idea about how paralysis would be avoided in the event of political cohabitation.

The decisive factor, however, was less the substance or even the dire communication of the idea, but rather its source. Electors may not like the political system, but they are even less keen on the politicians who operate in it. So when the latter come up with plans to overhaul the former, these are not regarded as straightforward proposals for reform, but schemes shaped by ulterior motives. We saw that first with John Prescott's dream of a north-east assembly, then with Nick Clegg's hopes for the alternative vote and now once again with David Cameron's push for mayors. All three promised more power for the ballot box, but all three have fallen victim to it, through plebiscites. Even Bristol, the one city that leaned the other way yesterday, may have done in reflexive reaction against the local Lib Dems, who campaigned for a No.

Engineering this sort of an electoral double negative would be unattractive, even if it were feasible. But securing reform through straightforward success in referendums in which voters pass judgment on all sorts of issues not on the ballot appears a lost cause. Regarded as products of a failing system, politicians are deemed too tainted to fix it. That leaves representative democracy in a sclerotic bind. Any party serious about forging a new politics must now consider putting specific plans in its manifesto, securing a mandate for them in the traditional way – and then simply getting on with it.