Why is it so difficult to edit the New Statesman? Since Kingsley Martin retired in 1960, after 29 years in the chair, the magazine has bid hail and farewell to 12 different editors. John Kampfner, who left last week, lasted two years and 21 months. Some of his predecessors served less than that. I hope it will not seem unduly immodest of me to mention that I lasted seven years, longer than anybody since Martin. But I was fired in the end. Not all the others suffered the same fate, but most left office with a sense of disillusion and disappointment and, in some cases, mental turmoil.

Paid-for circulation, which touched 90,000 in the 1950s, now bobbles around in the low 20,000s. Every editor starts with a belief that he (it's always been a he so far) can restore the magazine's glory days. Every editor finds the circulation stubbornly resists more than brief upward movement. Kampfner presided over a handsome redesign, strengthened the magazine's political authority, won a string of awards, got several dramatic scoops, and achieved a high media profile both for the magazine and himself. (I should add that, since leaving the editorship in 2005, I have continued as a columnist.) But though headline circulation rose to 30,000, that included some 5,000 of what the Audit Bureau of Circulation politely calls "monitored free circulation". Even the paid-for circulation was boosted by a higher than usual (at least by NS standards) marketing spend.

The magazine's owner, Geoffrey Robinson, was enraged by mounting financial losses for, as he saw it, inadequate gains in sales. Kampfner was equally enraged when Robinson, to stem the losses, cut the marketing budget almost to zero. Actively purchased sales fell to just over 22,000, as revealed by last week's ABC figures. Whether Kampfner was sacked or walked is unclear, but by the end, the two were heartily glad to be rid of each other.

The New Statesman's problems are partly those of any small paper that is independent of large publishing groups. Even in its heyday, it was rarely profitable, relying on legacies from early Fabian benefactors. Robinson, who took over in 1996, has, at various points, "invested" some millions in the magazine. But while he is very rich, he lacks the resources of, for example, the Telegraph group, which doubled and then trebled sales of the Spectator and eventually created the virtuous cycle - revenues that are sufficiently strong to support continued marketing while also returning a profit - that Robinson seeks. Nor can the New Statesman draw upon the back-office support, distribution muscle and free national newspaper promotion that the Telegraph delivers to the Spectator.

And a leftwing magazine - even one for the proverbial champagne socialists - struggles to attract the ads for fast cars and fine jewellery that help the Spectator's cash flow.

But there is more to it than that. The New Statesman has an identity problem which reflects an identity problem on the British (and perhaps worldwide) left. Conservatives, though their party has been out of power for more than decade, retain a strong sense of who they are and what they stand for. Paradoxically, it is only now, as they get a sniff of power again, that they are beginning to lose that confidence, and it is already, I think, evident in the Spectator's pages, if not its circulation.

The left, however, has struggled for more than 30 years and, if anything, the crisis grew more acute from the mid-1990s. Though Labour returned to power, it did so as new Labour, ruling on conservative terms. This was disorienting both to the magazine and its readers. The New Statesman was traditionally on the Labour left (though never the hard left) and its past was therefore part of the baggage the Blairites wished to bury.

Moreover, the magazine's traditional audience was composed, not so much of trade union and Labour activists (that was Tribune's territory), but of what used to be called Hampstead intellectuals. They were academics, senior civil servants, lawyers and other professionals of a liberal/leftish cast of mind. Their great causes were based on opposition to colonialism, racism, hanging, arts censorship, and laws restricting homosexuality, abortion and divorce. Those battles have been largely won. The Hampstead intellectual is an anachronism.

Successive editors have tried to establish a new identity for the New Statesman. One went for muckraking investigations; another wanted it to be a magazine of the streets rather than the corridors of power; a third wanted to make it the ultimate insiders' paper, providing unique insights into new Labour. I attempted, at least initially, a Spectator of the left; urbane, amusing, irreverent, elegantly written, and as cussedly leftwing as the Spectator was rightwing. Kampfner, with his deputy Sue Matthias, created something that, for the first time, actually looked and behaved like a magazine.

Some of these approaches had more merit than others. Some editors, no doubt, were better than others. But none had the long-term marketing support necessary to establish a new identity in the public mind. To many potential subscribers, the NS is still a quaint relic of the 1930s and 1940s. Robinson, it seems, is still agonising over Kampfner's successor. Matthias (who has worked at the Observer and the Independent on Sunday and is one of the most highly rated editors in the industry) should be a strong candidate. Steve Richards, the Independent's political columnist and a former NS political editor, who has long been courted by Robinson for the top job, is another possibility.

More important perhaps is the future ownership of the New Statesman. Robinson, despite being a Labour MP and ally of Gordon Brown, is an impeccably hands-off proprietor in editorial matters. Though he seeks the broad approval of the Brown inner circle when new editors and political editors are appointed, he has preserved, and even enhanced, the magazine's independence.

It remains the most intelligent and politically astute magazine on the market and still attracts star writers at remarkably low rates. But Robinson's experience of publishing is limited to the NS, which, as a business, he runs rather like a family corner shop. He has little understanding of the industry, and doesn't always heed advice from those who have more.

Yet it is not clear that there is any credible alternative. No large publishing group has shown much interest. The Guardian Media Group has occasionally discussed a takeover only to conclude its energies would be more profitably expended on a few more motorcycle magazines. Potential minority shareholders are wary of joining forces with Robinson. An even wealthier sugar daddy is always a possibility, but there is the risk that he or she, unlike Robinson, would wish to dictate the political line.

So whoever succeeds Kampfner needs inspiration (and a touch of genius), patience, stamina - and a willingness to cause outrage which, for a small magazine, is the best way of getting noticed. The moment ought to be a propitious one. The demise of Thatcherism and the sudden collapse of American conservatism suggest the time has come for a left revival. Whether Robinson has the vision to seize the moment - or will appoint an editor capable of implementing it - is another matter.