'That the Tories hate him so suggests Balls is one of the few Labour figures they fear.' Photograph: Christopher Thomond for the Guardian

One hesitates to challenge so august an authority as Anthony Seldon – fearing a visit to the headteacher's study, as it were – but his call today for Ed Balls to "fall on his sword" is so wide of the mark, it needs to be taken on.

Seldon, who doubles as a political biographer and master of Wellington College, has written an open letter to the shadow chancellor, published in the New Statesman. It sets out why it would be best for Ed Miliband, the Labour party, Balls himself and his wife Yvette Cooper, if Balls were to quit frontline politics until, say, 2017. Seldon suggests Balls could use the next four years following Seldon's own line of work, writing a biography of George Brown or teaching in a school. Or perhaps studying for an MBA. Anything but Westminster.

The master reckons this will rid Ed Miliband of the stale breath of the Brown era and the stench of its hardball tactics. It would create room for the return of David Miliband and Alistair Darling – though which one should get Balls's job, Seldon does not quite say. It would make Labour more appealing as a coalition partner to the Lib Dems, should parliament be hung again in 2015. It would allow Cooper a clear run. And it would be better for the Balls-Cooper children to have their dad around more.

Put aside the cheek of Seldon advising Balls on how to bring up his kids. (Presumably if Balls wanted that advice he could have paid Wellington's sizeable fees to get it.) Seldon is wrong on the politics too.

"Economic credibility would be more readily restored with your departure," he tells Balls, adding: "Your critique of the government's austerity strategy may never win back public trust and your proposals for the economy will never convince." But what economic credibility Labour now has it owes largely to the shadow chancellor, for the simple reason that he called it right when so many others called it wrong. In August 2010, he delivered a speech at Bloomberg's London HQ, which broke the consensus of the time, explaining that austerity would not nurture recovery but choke it. Some laughed off his warning of a double-dip recession. But Balls – a first-class economist before he was a politician – was right and they were wrong. As I've argued before, Balls is one of the very few people in politics able to utter those golden words: I told you so.

As George Osborne finds it ever harder to generate growth, as he presides over borrowing that has swollen not shrunk, as previous allies, including the IMF, suggest the austerity medicine is not working, Balls becomes ever more vindicated. Asking a politician to resign when they get things wrong is one thing. Demanding they quit when they get things right is a kind of madness.

Seldon might perhaps ask himself why it is David Cameron turns a shade of puce every time he finds himself facing Balls. Why is it the Tories hate him so? In politics, such loathing is a compliment. It suggests Balls is one of the few Labour figures they fear. The same goes for the right-leaning commentariat's regular demand that Balls go, a chorus Seldon has now joined. No one ever demanded the head of Gavin Strang.

Curiously, Seldon also mentions Europe, casting Balls as an opponent of a referendum. That will come as news to the Balls-ites in the shadow cabinet who were said to be agitating for Miliband to pre-empt Cameron and call for a Europe plebiscite. It also draws attention, in a way unhelpful to Seldon's argument, to Balls's Europe record. As the biographer surely knows, the loudest voice in New Labour's inner councils against joining the euro always belonged to one Ed Balls.

But one does not have to be convinced of Balls's talents to oppose his departure. Many have been surprised and impressed by the degree of Labour unity since 2010. Most of the credit for that belongs to Ed Miliband. But it's also partly a function of the fact that the powerful Balls camp feels represented at the top table. Exile Balls and there will be a sizable group that believes it lacks a voice. Resentments will grow. Call it a team of rivals, pissing out of the tent or keeping your enemies closer – the idea is the same. It's best for Ed M to have Ed B on board.