The Home Office has been a graveyard for ministerial ambitions for decades.
Michael Cockerell finds out why.

The Home Office has long had a reputation as a glittering coffin. Many recent home secretaries have, as one of them put it to me, left the place feet first. There have been six New Labour home secretaries in the past 13 years – three of whom were forced to go.

To try to capture the culture and DNA of the most hazardous department in government for a new TV series, The Great Offices of State, I have had access to the latest Home Secretary, Alan Johnson, and his Sir Humphrey, David Normington, and I have talked to many previous home secretaries and their normally camera-shy top mandarins.

Nearly 30 years ago, the Home Office moved into new, specially built headquarters. Its brutal concrete and glass exterior seemed to reflect the distinctive culture that had grown up among its civil servants – secretive, defensive and designed to repel boarders. Sir Hayden Phillips, a long time Home Office mandarin, says: "There was a sense of being bunkered – always being got at and not being understood – with people not appreciating you were trying to do a good and honest job."

New Labour's first home secretary, Jack Straw, says that he and his officials would refer to the Home Office building as if it were a KGB jail. "We used to call it Lubyanka. It was like a prison and with little cells. As home secretary you could go up in the ministerial lift – you had your own loo. And with a bit of luck you'd never be contaminated by any other form of human life – except in a meeting. And later you'd go down in the lift and leave."

The Home Office has long had the reputation of being a backward-looking and closed society where the policies would continue unchanged, whoever came in as home secretary. Michael Howard says that when he was the last Tory home secretary journalists would ring the Home Office to ask about policy and would be told by officials: "Well, the Home Office policy is this – however, the Home Secretary thinks…"

Howard had come to the Home Office with a hard-line agenda to cut crime. But he came up against the Home Office's own distinctive view that had developed over the years. "I was shown charts which showed crime rising inexorably and the officials actually said to me: 'It is going to continue to go up and the first thing, Home Secretary, that you have to understand is that there is nothing you can do about it'."

Howard felt that in his four-year term he had managed to change that culture and he was saddened to hear what his successors had found.

"I hate to agree with Michael Howard about this," says David Blunkett, the former Labour home secretary, "but he and I shared the view that the Home Office didn't really believe that they could change the world – that they could really make a difference in reducing crime."

And Jonathan Powell, who was Tony Blair's chief of staff, recalls the presentation the Home Office gave to the New Labour Prime Minister: "They presented projected figures that showed crime inexorably rising during our years in government. And they said this was a result of the economy improving, and as the economy improved, there'd be more stuff to nick and therefore there'd be more crime.

"We were a bit bemused by this and I asked what would happen if the economy turned down and we suffered a recession rather than a growing economy? They said: 'Oh well, crime would go up, there'd be more people to nick things'."

The former Cabinet Secretary Lord (Richard) Wilson, who was the Home Office's top mandarin under both the Tories and New Labour, explains the official mindset. "In the Home Office, you do have a sense that you are the Department of Law and Order, in an age when authority may not be the most fashionable of the roles to play. They feel that faced with the growth in crime, this is the sum total of endless other failings which no one home secretary has the capacity – however brilliant and however powerful – to tackle."

Blair's first home secretary was Jack Straw, but over the years the PM came to feel that Straw had fallen prey to a defeatist Home Office orthodoxy on crime. "I did get irritated with Tony sometimes and used to say to him: 'I think you're asking me to push water up hill, aren't you?'," Straw says.

Blair chose the populist figure of David Blunkett to replace Straw. But the new home secretary came up against the new top mandarin, Sir John Gieve. He believed that Blunkett was more driven by getting good crime-fighting headlines than by hard analysis. "David Blunkett wanted to shake up the Home Office," Sir John says. "The political agenda was to seize this traditional bit of Tory ground and reclaim law and order as a Labour issue. That made the Home Office the main political battlefield."

Blunkett admits: "John Gieve and I did have our clashes and frustrations. But what I needed was a can-do attitude." Sir John counters: "I was responsive to David's policy agenda but he should have been looking to me not just to be a supporter, but to be someone who was telling truth to power."

Blunkett came to feel increasingly frustrated at the Home Office. And as his private life became entangled in official business he felt one day that even the run-down building itself was in revolt against him. "I went in one morning and the building was creaking, it was beginning to fall apart, a kind of metaphor really," says Blunkett.

"I walked into the upstairs toilets and I was very quick on the uptake in realising there was something amiss here. And I was very glad I did, because I called in one of the private secretaries: he declared with an entirely straight face that the sewage system had obviously gone into reverse and the whatsit was now in the washbasin and the bath area rather than down the toilet. And I thought, well that just about sums it all up."

After Blunkett resigned, the Home Office moved to shiny new hi-tech headquarters. Charles Clarke was the new Mr Fixit. But he become embroiled in a huge political row when it was revealed that the Home Office had, over six years, released a thousand dangerous and violent foreign prisoners – including murderers and rapists – without them being considered for deportation. The Home Office had no idea where they were, nor how many more such cases there were.

Charles Clarke says the scandal was the result of the "absurd and ridiculous system" where one part of his sprawling empire did not know what the other part was doing. And in our film he takes the highly unusual step of publicly pointing the finger at his top mandarin. "I made the mistake in relying on Sir John Gieve, who had assured me that he'd got the matter in hand in terms of dealing with the immediate situation – which turned out not to be the case."

Sir John responds: "We should have dealt with the thousand cases better over six years – actually, before I was there as well as when I was there. But obviously Charles Clarke had doubts about me, and" – Sir John added, after a lengthy pause – "I about him."

Sir John had left the Home Office by the time Charles Clarke was, as he puts it, "sacked" by Tony Blair over the foreign prisoners fiasco. The new home secretary was the Glaswegian hard-man John Reid.

Sir David Normington, who took over from Gieve and remains Permanent Secretary, says: "John Reid came in like a whirlwind. He was put here to steady the ship and take a grip." Sir David was at Reid's side in the Commons when the new home secretary famously described his division dealing with asylum and immigration as "not fit for purpose".

"I agreed with his analysis," says Sir David, "but wish he hadn't used the 'not fit for purpose' phrase, because of course that then became the label that attached to the Home Office for the next several years and to some extent still does when we run into heavy weather."

But what, I asked, had the phrase done at the time for morale in the Home Office? "Morale was low because the previous home secretary had lost his job," says Sir David. "That was the result of something that we had done wrong. We're here to serve our home secretaries, not to cause problems for them. Departments can lose their reputation in a moment and on that day our reputation was absolutely in shreds."

Over the past four years, Sir David has done his utmost to restore the Home Office's standing. Although he had strong personal reservations, he went along with the plan hatched by Tony Blair and John Reid for the biggest shake up in the Home Office's history. The department was split into two ministries on continental lines: criminal justice, prisons and probation were handed over to the newly created Ministry of Justice, and the Home Office became effectively the ministry of the interior, responsible for internal security, law and order, immigration and terrorism. That decision has led many former secretaries and leading officials to wonder whether the Home Office any longer deserves its designation as a great office of state.

Labour's latest Home Secretary, Alan Johnson, has no doubts that the Home Office is still one of the great departments: "And it had better stay one now I'm in it." But any future incumbents would be wise to heed the words of a beleaguered home secretary who wrote in a memo to his Sir Humphrey 50 years ago: "Poor old Home Office, we don't always get it wrong. But we always get the blame."

'The Home Office', the first of Michael Cockerell's three-part series on 'The Great Offices of State', is on BBC 4 at 9pm tonight.