It's not that voters are apathetic or indifferent or lazy. It's that they know from long experience that the identity of the minister in charge makes only the slightest dent in the policy of the department.

Take what seems to be the main talking point for journalists: the replacement of Ken Clarke by Chris Grayling as justice secretary. It's a "lurch to the right", say pundits, excitedly. (Why, incidentally, to parties only ever "lurch" to the right? When did you last hear of someone "lurching" to the centre?)

No one outside Westminster, though, will expect anything to change. The supremacy of the European convention on human rights, like the preference for non-custodial sentences, is regarded as an intrinsic part of the system, something beyond the capacity of any minister to alter.

Every home secretary comes to office promising to deport dangerous fanatics, just as every foreign secretary promises to stand up to Brussels, every business secretary to "cut red tape", every chancellor to keep taxes down. It never happens. And the reason it never happens is that the minister is encased in a bureaucratic machine bigger than he is. There is little question that he wants these changes to occur – he can read opinion polls as well as anyone else – but he is stabbing at buttons that are disconnected, tugging at levers that have worked loose.

In real life, Sir Humphrey will almost always get the better of Jim Hacker. Where Hacker has a single special adviser, Sir Humphrey can call on hundreds of full-time professionals. Where Hacker is passing through – and already half-thinking of his next job – Sir Humphrey is there to stay. Where Hacker is distracted by Commons votes, constituency casework and after-dinner speeches, Sir Humphrey is in the department full time.

With enough patience and tenacity, Hacker can eventually make changes – but it takes years to overcome the inertia of his permanent officials. Michael Gove is reforming education, and Iain Duncan Smith, like James Purnell before him, is reforming welfare. Both men, understanding how the system works, knew they had to stay at their posts to complete their work.

Some Comment is free readers might at this stage be thinking that Sir Humphrey is a useful ally. Well, on some issues, he is, notably penology and education. On other issues – public spending, the American alliance, nuclear power, airport expansion – he is more of a nuisance to the left. And on quite a few, he is at odds with almost the whole country: it was a civil servant, for example, not a minister, who committed Britain to bailing out the euro, despite our non-membership.

The problem is not this or that policy outcome; it's the delegitimisation of representative government. A minister is supposed to be there to remind his permanent officials that they work for the rest of us. If, instead, he becomes the cabinet champion and public spokesman for his department, democracy is vitiated.

In consequence, the system loses its legitimacy, elections become devalued. "What's the point in voting, nothing ever changes?" people say. And, by and large, they're right.