Once again the Conservative party has proved why it has a PhD qualification in political ruthlessness, while at the same time the Labour party is struggling to even manage a GCSE retake. When most of us were cleaning our teeth this morning, Boris Johnson was still the bookies’ favourite to win the Tory leadership and succeed David Cameron as prime minister. Yet by lunchtime Johnson – I absolutely refuse ever to call him just by his first name in print – was a political corpse with Michael Gove’s lethal stiletto between his substantial shoulder blades. How ghoulishly appropriate that today is the anniversary of Hitler’s Night of the Long Knives.

You simply have to hand it to Tory MPs. They are professionals. A generation ago they took down Margaret Thatcher, disposing of her in a way that the Tory grassroots would never have allowed but which enabled John Major to win a general election. Now they have taken down Johnson in order to ensure that a new generation of credulous Tory activists don’t put the party’s election chances at risk by selecting him as leader. An already ghastly week for Labour just got a bit worse as a result.

The Johnson bubble was always going to burst. Some of us said this long ago. The only question was whether it happened before or after his leadership bid. Luckily for the Tory party, they made it happen now. Everything about the supposed public appetite for Johnson as leader of the nation was potentially damaging for Johnson himself, for the Tory party, for the country and for politics. He would have been a disaster.

Where does one start? With the flawed character himself, perhaps. All politicians have ego, but Johnson is a narcissist. He’s a lightweight, a first-degree self-publicist with a second-rate mind. He has a shabby back story, with the truth, with details, with responsibility and with women, any one of which could have ruined his prime ministership if he had been allowed to get that far, and may have had something to do with what happened to him on his way to work.

Nor, echoing Clement Attlee’s terse retort to a sacked minister, was he up to the job. He is not serious. He is essentially frivolous. He played cricket when he had led the country on to the rocks. He can’t negotiate, as anyone who watched him in London will attest. Theresa May, who is almost everything that Johnson is not, skewered him on that today.

More immediately, Johnson made a last-minute jump into the leave campaign, loving the attention that his indecision attracted from a press that has shamed itself by fawning on him so much. Then, having won the campaign for leave, he seemed three days later to be preparing the ground for a complacent dash back towards a sort-of remain. Two weeks ago migration was a problem. This week it wasn’t. It was treacherous to the leave side and humiliating to remain. As ever, it was all about him.

Most important of all, though, he isn’t half as popular as the Tories think. Johnson won in London because he was facing Ken Livingstone. But he wouldn’t have beaten Sadiq Khan if he had run again. Now he has dug up his own pitch by taking Britain out of Europe against London’s will, so Tory chances in London would not have soared under him. If the Tory party’s voting rules were as lax as Labour’s, a lot of people in the City would have joined the party simply to vote against him.

Then there are the polls. The Mail on Sunday said that May headed Johnson among Tory voters, which doesn’t square with Johnson’s lazily assumed popularity. A Times poll today even put May ahead among Tory members, the people with the power in this contest. The fact is that the Johnson brand was turning pretty sour. As Cicero, whom Johnson is fond of quoting, might have put it: how long, Johnson, do you intend to go on abusing our patience? Yet not even Catiline visited such troubles on Rome as Johnson has now contrived to visit upon Britain.

Even Gove had had enough, perhaps prompted by his wife, perhaps by his advisers, perhaps by his own ambition, perhaps by Rupert Murdoch, to turn on Johnson. Talk about last minute. This was a road to Damascus conversion taken as Gove got into downtown Damascus itself. Gove’s sole acknowledged reason for running, according to his announcement, was that Johnson could not do the job. Politics doesn’t get more personal than that.

Johnson’s removal from the race transforms all calculations. Yet it need not mean that the Tory party is in longterm chaos. The party may be ruthless, but it is also curiously loyal. Not one of the candidates who were finally announced at lunchtime proposes to defy the result of the referendum. Although two of the candidates, May and Stephen Crabb, were remainers, all appear genuinely willing to own the outcome.

Johnson’s eclipse makes a May versus Gove contest in the final round likely. In the past, May’s chances tended to be dismissed because, in Westminster terms, she is like Kipling’s cat that walks by itself. She rarely works the room or the studios. She frequently does her own thing, which made Cameron suspicious. Though her leadership ambitions have never really been in doubt, she does not have much of a machine. The result is that she had relatively few committed supporters until now.

Nor is she an ideologue, as she said today. This may help in febrile times. The Angela Merkel comparison is an illuminating one, much more relevant than talk of a second Thatcher. May’s speech said the right things, especially about social justice and the abandoned working class. At present, these are just words. But she has made interesting speeches in the past, she offers competence and practicality – and she has built up some much-needed political dignity too.

If Johnson had turned on a sixpence and announced, as May did today, that he was no longer proposing to withdraw the UK from the European human rights convention and court, he would have been denounced for flip-flopping because that’s his reputation. That may yet happen to May too, but she has a bit more dignity in the bank. She did it to open her tent to liberal Tories, which is sensible, and if she wins it will have been a well-judged concession.

In May, the Conservatives have a candidate who today looked like a prime minister, behaved like a prime minister and sounded like a prime minister. What is more, after a truly convulsive day, she very possibly could be the next prime minister. We certainly need a much better one than the unlamented Johnson.