Jeremy Heywood on the main staircase at No 10 in 2011. He was a trusted adviser to many prime ministers during his long career.
Photograph: DPA/PA Images

Jeremy Heywood, Lord Heywood of Whitehall, who has died of cancer aged 56, was a player in all the major events that affected the nation’s destiny over the last three decades. The former cabinet secretary played key roles in the Black Wednesday financial crisis of 1992, the Iraq war from 2003, the financial crisis of 2008, the creation of the coalition government in 2010 and the current Brexit negotiations.

He also became a trusted adviser to successive prime ministers, managing to cross the divide not only between opposing parties but also between different factions within the same party. During his long career in Whitehall he was a trusted civil service adviser to Lord (Norman) Lamont, when he was chancellor under John Major in the 1990s, to both Tony Blair and Gordon Brown despite the poisonous relationship between them, and latterly to David Cameron and Theresa May, who were colleagues rather than friends.

He was, as even his critics admit, the most powerful person in the country of whom most people had never heard. He himself acknowledged this in evidence to a parliamentary select committee in 2012. “I like to be invisible,” he said.

Inside Whitehall he was far from invisible. He was more invincible, and certainly controversial. In the Whitehall battle of ideas, he was an ardent moderniser, wanting to get away from the stuffiness and arcane rituals of the system to get things done. He used all the latest management jargon, describing the civil service to foreign governments as “a net exporter of ideas”, ministers as “customers for our advice” and official reports as “a product”.

He also could be ruthless even at a young age. After the financial crisis caused by Britain plunging out of the Exchange Rate Mechanism in 1992, he implemented such savage cuts at the Treasury that civil servants referred to their offices as “the corridor of death”. The reforms have endured until today, with the Treasury being one of the smallest Whitehall departments.

Born in the former mill town of Glossop in the High Peak district of Derbyshire, he was the son of Peter Heywood, a teacher, and his wife, Brenda (nee Swinbank), an archaeologist. He was educated at Bootham school, a private Quaker establishment in York, where his father taught English and where he became head boy.

He went on to study modern history and economics at Hertford College, Oxford: his friends, according to contemporaries, tended to be leftwingers and anarchists, and at one stage he went off to Pakistan on an overland trip. He also became a chain smoker, and was an avid supporter of Manchester United.

On graduating, he chose Whitehall, starting at the Health and Safety Executive (1983-84) but quickly moved on to the Treasury. It was there that he began to shine as one of the sharpest minds in Whitehall. He was private secretary to Lamont (1991-93) and then, only just over 30, worked closely with Lamont’s special adviser, the future prime minister David Cameron.

He went on to become private secretary to the new chancellor, Kenneth Clarke, and in 1994-95 drew up his fundamental review of the Treasury’s structure, which became a blueprint for other changes.

When Blair became PM in 1997 he summoned Heywood to his office and made him economic and domestic secretary, which meant he became one of the key figures bridging the Blair-Brown divide in the new government. To his credit – and despite the acrimonious battles between the PM and his chancellor – Heywood was both trusted by Blair and by Brown’s rising star, Ed Balls. In 1999 he was promoted to principal private secretary to the PM.

Then in a rare move Heywood, who had also married a fellow civil servant, Suzanne Cook, in the year Blair won the election, left Whitehall. He became managing director of the US bank Morgan Stanley in 2003 and decided to have more time with his newborn son.

But it was not to last. By 2007 he had been lured back to Whitehall by Brown – first as head of domestic policy and strategy at the Cabinet Office, and within a year as permanent secretary in Brown’s office. He was immediately thrown into the financial crisis, which stretched his ability to the limit.

When the coalition was formed after the indecisive 2010 general election, Cameron kept him on at Downing Street, promoting him first to cabinet secretary in 2012 and combining the role with the head of the civil service in 2014.

The coalition years were dogged with problems from the controversial appointment and later resignation of Andy Coulson, the former News of the World editor, to No 10 to the appointment and then resignation of Steve Hilton, Cameron’s guru. Hilton was first wooed by Heywood, but after he proposed the sacking of 90% of all civil servants, effectively dispatched by him. When asked about his relationship with Hilton by MPs, Heywood replied dismissively: “Steve has now left for California as far as I am aware.”

The Brexit vote in 2016 led to Cameron’s resignation and the arrival of Theresa May. The loss of her overall majority in the 2017 election kept Heywood carefully balancing the different factions of the Tory party, using his skills to protect the PM and trying to keep Brexit on track. Indeed practically his entire Whitehall career had been built on keeping the various political factions – mostly inside political parties – together and stopping them derailing the business of government.

He was not uncontroversial. The right attacked him for changing the face of Whitehall, the left for shielding the PM. He was not a fan of freedom of information when it interfered with the business of government – and was very hostile to the disclosures by the former National Security Agency adviser Edward Snowden, telling the former Guardian editor Alan Rusbridger that the paper could become a target for “our guys” in British intelligence for publishing the disclosures.

But he will be remembered as a brilliant civil servant who could juggle 16 things at once and a very clever negotiator who was able to smooth the path of government amid warring political factions. If ever he had written his memoirs they would have been full of amazing disclosures.

After his diagnosis of cancer last year he took a leave of absence for treatment. At the end of October he retired through ill health and was appointed to the Lords, his knighthood having come in 2012.

He is survived by Suzanne, a son, and a twin son and daughter.

• Jeremy John Heywood, Lord Heywood of Whitehall, civil servant, born 31 December 1961; died 4 November 2018