So President Obama has bitten the bullet – and General McChrystal had to go for his insolence and dysfunctional personal command, and insubordination. Obama took a leaf out of the book of his hero Abraham Lincoln, who sacked general after general in the American civil war until he found the right one.

Honest Abe was as tough as old boots, and he sacked primarily because he thought the generals incompetent. The problem with McChrystal is that the arrogance of his special forces team that formed his headquarters couldn't hide that they just weren't delivering enough on their new counterinsurgency strategy which was to bring the fighting to an end within a couple of years.

This is the most dramatic sacking of an American commander by a president since President Truman fired Douglas MacArthur because he wanted to nuke the North and their Chinese allies in the Korean War some 60 years ago.

Following his month-long assignment to follow General McChrystal Rolling Stone reporter Michael Hastings concluded that: "Obama lost control of the Afghan policy about a year ago," as he told a radio interviewer. With this sacking, the president is trying to grab back the policy joystick on Afghanistan.

By appointing General David Petraeus to the Afghan job he has someone with a proven track record. He was the architect of the so-called surge in Iraq, which brought levels of violence down dramatically, at least enough to allow the Americans a semblance of an exit strategy. He is also the real architect of the counterinsurgency doctrine McChrystal preached – indeed he had been, and was until a few hours ago, McChrystal's superior officer.

Above all, Petraeus has shown a lot more political savvy than his erstwhile pupil. He knows how to work with Congress and how to work with politicians in the field. Part of the problem, which was identified in the Rolling Stone piece, is that McChrystal tried to run the military operation and the political one at the same time. Meanwhile his staff disparaged senior international diplomats that might get in the way. This week Britain's senior negotiator Sir Sherard Cowper-Coles quit because he thought the McChrystal plan couldn't work.

Last year McChrystal called for first 21,000 and then 40,000 extra US troops to make his concept to defeat the Taliban work. Obama so far has given him 51,000 more troops. McChrystal said the international force of 150,000 would allow his troops to defend the civilian population, build up the Afghan army and police, and sufficiently thwart the Taliban to force them to the negotiating table.

So far, not so good. The first big operation this spring – to take, hold, and redevelop the fertile poppy growing district of Marja in Helmand is still far from successful. The Taliban at first lay low, went home for the poppy harvest, are now back in action raiding farms and killing and threatening the farmers. McChrystal himself has described Marja as "a bleeding ulcer".

This month American, Afghan, British and Canadian troops were supposed to begin the next major operation to restore order and government to Kandahar. This was to be the litmus test of the whole McChrystal strategy – at least for 2010. President Karzai, who has spoken up for the general – grounds for suspicion enough – has prevaricated about final approval for the Kandahar mission. The most powerful man in the town is his half brother Ahmed Walid, who is a law unto himself. So the week before last it was quietly announced that the operation was postponed until the autumn at the earliest.

So the McChrystal operational plan is stalled. Out on the ground McChrystal was losing the confidence of a lot of the American and British soldiers on the ground because of his instructions for "courageous restraint". These rules mean a soldier can only fire back if he can clearly indentify the figure of the man who fired at him, in order to minimise the risk of hitting civilians. This endangers the lives of soldiers because it means you cannot fire back in ambush positions – where the standard procedure is to "put down fire", a curtain of ordnance, in order to beat a retreat.

This of course hasn't affected the special forces, who became an informal Praetorian Guard for McChrystal and his command cell: they increased the number of special force squads to 19 recently. The critics accuse them of doing too much shooting and too little negotiating and reconciling with the Taliban. American special forces are accused of a number of nasty killings of civilians round Kandahar this spring, including shooting two pregnant women and a passenger on a commuter bus.

There will be a lot of sucking of teeth now and quiet rethinking about where to go in Afghanistan, not least among British commanders who had a particularly close relationship with McChrystal.

Obama has laid down the gauntlet to the generals – they have to explain themselves better to him and the American public. Before this, they tried to intimidate him, by all accounts, as the men who knew the business in a subject area of which he knew little. The same process should happen here with arrival of the new coalition government. To argue for more of the same, that Afghanistan is vital to the security of mainland Britain, and that there is no other option is not good enough.

Yesterday another Royal Marine was killed in Sangin, where British soldiers have been trying to hold the town for the past four years in an area surrounded and infested by Taliban. Some 100 of the 303 British fatalities in Afghanistan have been recorded there, and about 1,000 wounded. Our generals say that not much is likely to change there, because in the McChrystal thinking, "Sangin is not on the main effort". But as one of the main entrepots on the east-west heroin trading route from Kandahar into Iran, it sure is on somebody else's main effort.

The most chilling aspect of the Rolling Stone article is that no one in it talks of victory or success, even. "The facts on the ground are not great," says Celeste Ward of Rand, a McChrystal adviser, "and are not going to become great in the near future." Another adviser, Marc Sageman, formerly of the CIA, adds: "Afghanistan is not in our vital interest; there's nothing for us there."

For the British military, especially the British special forces, McChrystal was a hero of almost Homeric proportions. His dismissal should make the commanders, diplomats and politicians think hard and think again about the Afghanistan policy from top to bottom. It is no use them clinging to the notion that the British army needs to defend its military honour and prowess to prove Britain is still a vital ally to the US – which is how some argue for our troops still being there. Notions of honour and fidelity are not in any sense practical operational objectives.

The McChrystal episode is not just a matter of an exasperated president sacking one of his generals for cheek. The whole debacle, the stalling campaign, the wavering focus, and the weird way it was all revealed to Rolling Stone, demonstrate the odd one-dimensional quality, very common in the special forces milieu of Stanley McChrystal.

From being the man of the hour last year, he now looks a bit like the wrong general with the wrong plan in the wrong place at the wrong time. A friend who knew McChrystal well in the special forces has just texted: "McChrystal's virtue is his vice: military obsession at the expense of breadth of vision. Wrong man for a profoundly political job."