The president had spent the weekend interviewing several generals to replace the disgraced Michael Flynn, included the retired general Keith Kellogg, the former UN ambassador John Bolton, McMaster and the West Point superintendent, Lt Gen Robert Caslen. Late last week, Robert Harward, a retired vice-admiral and a former aide to the defense secretary, James Mattis, declined Trump’s offer to replace Flynn.

McMaster’s appointment follows a week of disarray over the direction of the National Security Council, after Flynn was forced to resign over revelations that he had lied to Vice-President Mike Pence about his calls discussing easing sanctions with Russia’s ambassador.

However, much like Trump’s pick to head the Pentagon, James Mattis, McMaster is a known entity to traditional US allies, and a figure reassuring to the US security establishment Trump has often scorned.

During the announcement at Trump’s Mar-a-Lago club in Florida, McMaster thanked Trump for the appointment and looked “forward to joining the national security team and doing everything that I can to advance and protect the interests of the American people”.

Yet it remains to be seen whether McMaster’s imminent arrival at the White House will settle an ongoing struggle for the future course of national security and foreign policy. At least one other candidate for the job, McMaster’s ally David Petraeus, dropped out of the running after insisting on the independence to select his own staff.

Trump and McMaster at Mar-a-Lago on Monday. Photograph: White House Press Office

Petraeus praised McMaster’s appointment without qualification.

“HR McMaster is a brilliant officer – a battlefield hero and a true soldier-scholar who will serve President Trump and the country exceedingly well,” Petraeus told the Guardian.

McMaster is expected to select his own NSC staffers. But one of McMaster’s friends, Petraeus’ former executive officer Pete Mansoor, said that as a serving military officer, McMaster was not in a position to attach many preconditions to his service.

“I don’t know if there were any conditions attached or not, but someone serving in uniform obviously has less leverage over a president than a retiree who can say no and get on with his life,” said Mansoor, a retired army colonel who teaches military history at the Ohio State University.

Mansoor also warned that McMaster would face numerous institutional challenges upon his arrival at the White House.

“He will do a fantastic job as national security adviser given the constraints under which he’ll have to operate. There’s obviously an alternative center of foreign policy and national security decision-making in the White House in the form of Steve Bannon’s Strategic Initiatives Group,” Mansoor said, referring to a new and parallel White House power center that NSC officials are eyeing warily.

“My hope is that HR provides another voice of reason in an administration that could use some good advice,” Mansoor said. “He’ll be a natural ally of Secretary of Defense Jim Mattis and the two will work closely with one another.”

In military circles, McMaster’s name is synonymous with a rejection of received wisdom that Trump has shown little willingness to tolerate and the Iraq war that Trump portrays as a symbol of foreign policy folly.

As a mid-career officer, McMaster wrote a scathing study of the joint chiefs of staff during the Vietnam era. His caustic thesis, published when he was a major, held that the most senior generals and admirals opted against declaring the war unwinnable or requesting a massive troop buildup, all to placate White House appetites for a limited war and pursue parochial priorities for their respective branches of the military.

Published under the searing title Dereliction of Duty, it was the first of several decisions in McMaster’s career that were expected to end it.

The most famous came later, with McMaster’s approach to the north-western Iraqi city of Tal Afar, on the Syrian border.

McMaster arrived in Tal Afar in 2005 as a colonel commanding the third armored cavalry regiment tasked with controlling a staging ground for cross-border attacks and an entrenched insurgency. At the time, the prevailing US approach to the occupation of Iraq was to attack insurgent positions as necessary and return to their mega-bases when not.

Effectively, McMaster reversed that approach and created a template for what would become the Iraq surge. In a well-documented campaign, the third ACR ringed the entrances to the city and treated the locals as an at-risk populace to protect, rather than a shadowy enemy one step removed from insurgents themselves.

Patrolling the city and partnering quickly with an Iraqi mayor and police, McMaster conceded American faults in the war and was quick to credit Iraqi politicians with successes. He was willing to meet with local potentates tied to the insurgency – “We understand why you fight,” he was once quoted as saying.

In contrast to Trump’s stated enthusiasm for torture, McMaster ordered detainees be treated humanely, and even polled detainees on how well the regiment followed through. And in contrast to the dark view of Islam put forward by Trump, his strategy chief Bannon and, most notably, Flynn – who once tweeted that fear of Muslims is “rational” – McMaster ordered his troops not to use derogatory terms to refer to Muslims.

On the backs of arduous fighting as well as nimble diplomacy, McMaster’s approach quieted Tal Afar. McMaster’s time there became a proof-of-concept for Petraeus and the rising band of counterinsurgent theorist-practitioners – currently out of vogue – to pursue the Iraq surge.

But though McMaster won acclaim, the army did not appreciate McMaster opting for a different approach – one that contradicted and upstaged its advice on the war – and it took Petraeus’s recommendation to get McMaster promoted to general. His subsequent service within the army’s training and doctrine command is institutionally respected but lower in profile, a move observers tend to interpret as lingering army discomfort with McMaster.

Their ally, Senator John McCain, who does not conceal his contempt for Trump, also praised McMaster’s appointment on Monday.

“I give President Trump great credit for this decision, as well as his national security cabinet choices. I could not imagine a better, more capable national security team than the one we have right now,” McCain said in a statement.

Kellogg will serve as McMaster’s chief of staff. Previewing a possible future appointment, Trump also said during Monday’s announcement that his administration will be asking John Bolton, a hardline senior diplomat in the George W Bush administration, “to work with us in a somewhat different capacity… He had a good number of ideas that I must tell you, I agree very much with.”