Say what you like about those ex-cabinet secretaries, they know when to speak out bravely against a government that is exceeding its historic powers, when it's on its last legs and they are all safely retired to the boardrooms and Oxbridge colleges where they can eke out their inflation-linked pensions in comfort.

If you missed it, what Watt's story said was that Labour has abandoned cabinet government and increasingly bypassed civil servants during its 12 years in power. I'm sure it's true, also sure that the same charges were made against previous prime ministers, notably Margaret Thatcher. "Presidential" is the charge they used. They used it against Harold Wilson too.

Lords Turnbull, Wilson, Butler and Armstrong, are the mandarins who delivered this verdict in evidence sessions and a memo to the House of Lords constitution committee. They did so in July, but no one saw fit to draw the media's attention to their thoughts.

I have met them all over the years and can even claim to know one or two of them, albeit slightly. Senior civil servants do not get to the top of Whitehall's greasy pole by befriending the hacks, especially those – there are some – not already predisposed to adore them for the nobility of their calling.

Nice chaps all, though with the possible exception of Lord Armstrong (cabinet secretary from 1979-87) I rarely felt myself in the presence of greatness, the kind of charisma which some great public servants once exuded. The current occupant of the post, Sir Gus O'Donnell, is modest to a fault and would shy away from any such label.

What I do on these occasions is turn up the memoirs or, in this instance, Hugo Young's Diaries (£30, Allen Lane 2008), to see what was said at the time. Particularly relevant seems to be the barrister Anthony Lester's remarks to Hugo on 12 October 1997.

Lester, himself a Lib Dem peer who fell out with Jack Straw this year, likens New Labour's victory to Singapore, a state which has eliminated the opposition and is already "building an autocracy" – a shallow judgment worthy of a columnist.

But it is cabinet secretary Butler's version (according to the noble lord and QC) that is worth recording here. No talk of autocracy from him, the new system is (reportedly) "wonderful", government by concentric circles in which Lord (Derry) Irvine, Blair's old boss whom he made lord chancellor, and Peter Mandelson are "the only ones that count". They are both clever – a delight.

"Butler also says that the cabinet committee system is a dead duck. Forget all that .... It is all run by the innermost circle. Ministers do not count for very much at all. Irvine in particular gets it all done … " etc etc.

With hindsight (indeed without it) this is mostly tosh too, not least because it makes no mention of one minister who has no intention of not counting – quite the opposite. Gordon Brown staged a coup of his own against Treasury officials – replacing the permanent secretary with one more to his liking (Turnbull) – and waging war against No 10. Thatcher sacked her cabinet secretary too, by the way.

Only after retirement did Turnbull accuse Brown of "Stalinist ruthlessness".

As a footnote to the Lester diary note, Young adds his own note, which is closer to the mark: to the effect that "this government is despising the civil service because they see it as part of the Tory disaster … yet the civil service were overjoyed when Labour won … And Labour will need the civil service … when things get tough. It is very ill-advised."

It's all history now, but worth noting. Early in Young's memoirs he notes that Robert Armstrong is cleverer than his rival, John Hunt – "a ruthless and devious man". Armstrong is less devious and gains his way "by sheer efficiency and intellectual power. Of course all civil servants at the top are discreet and devious … "

Sir Richard Wilson got the top job after Butler was eased out of No 10 in 1998 as a moderniser who would go along with No 10's plans to streamline Whitehall – replace "feudal ministerial baronies with a Napoleonic system", as I recall Jonathan Powell, Blair's ex-FCO chief of staff boasting at the time.

Wilson rapidly disappointed them by dragging his feet. By 2000 he is telling Hugo Young that Whitehall is no longer the sole source of policy advice – as it was in his youth – but also that the creation of a prime minister's department would have a bad effect on the "independent power and accountability of secretaries of state".

This was sensible and has proved to be the case. But, as usual, right is not all on one side. Labour in 1997 was disappointed with the civil service – friends on the inside have told me so. But they have also told me that cabinet secretaries such as Andrew Turnbull were sidelined to a surprising degree – so their attack on "sofa government" (Lord Butler made it in his Iraq war inquiry) may contain an element of pique.

By chance I was reading Ferdinand Mount's entertaining memoir Cold Cream (£20, Bloomsbury, 2008) over the summer. Mount's background is poor-but-posh, which gives him many insights into policy. It was as a good speechwriter and columnist, however, that he was recruited to run Thatcher's policy unit at No 10 – despite being a notorious moderate.

He tells similar tales of defeatist civil servants and hopeless ministers. Though never a Thatcherite, he came to admire the Lady's courage and zeal, seeking to save Britain from terminal rot by sheer force of conviction. I sense it was this can-do attitude that Tony Blair inherited and tried to copy – in contrast to the consensual drift under nice John Major.

The point is that, yes, the civil service mandarins make excellent points at Labour's expense – just as the tide of public sentiment is running strongly the Tories way – but there is also a good case against them and their cautious, self-serving ways.

Elected ministers feel the voters' wrath against failure more sharply than officials ever do – even now when they are slowly being dragged into the daylight to give a better account of themselves.