With U.S. and Iraqi officials talking of the imminent defeat of the Islamic State in Iraq, President Trump faces a challenge to swing a deal that President Obama failed to nail down: an agreement with Baghdad to keep U.S. forces in Iraq after the fighting ends.

Negotiations between Iraqi Prime Minister Haider al-Abadi and U.S. officials in Baghdad on a new status of forces agreement began in earnest this month, senior U.S. officials said. The deal would outline the legal and diplomatic parameters underpinning a long-term U.S. military presence in the country, and avoid the power vacuum that critics say developed after American forces pulled out in December 2011.

The issue is sensitive in Iraq, and Mr. al-Abadi this month insisted that there would be no “combat troops” in Iraq once the Islamic State is defeated, even though both sides said talks on the long-term role of the U.S. military in the country continue. An official who traveled to Baghdad with Defense Secretary James Mattis recently said “nothing has been finalized.”

There are an estimated 7,000 overall U.S. military personnel in Iraq, including several hundred Special Forces fighters advising the Iraqi army in the siege of Mosul. At the height of President George W. Bush’s “surge” against insurgents before the rise of the Islamic State, there were some 170,000 American troops in Iraq.

A new status of forces agreement got an endorsement from a key Iraqi Kurdish leader Tuesday. Masrour Barzani, chancellor of the Kurdistan Regional Security Council and the son of Kurdish Regional Government head Masoud Barzani, said on a Washington visit that he and his father would back a deal allowing U.S. forces to remain in Iraqi Kurdistan after the fall of Mosul.

Iraqi officials said Tuesday that the Islamic State now controls just two districts in eastern Mosul, with a majority of the city now under Iraqi control after months of brutal fighting.

“We have always been in favor of having American troops in the country,” the younger Mr. Barzani said during a speech at The Heritage Foundation.

“We have to look beyond, at the day after Islamic State,” he said.

A possible Kurdish push for an independent state is just one source of instability for Iraq. The Mosul fight has inflamed Sunni-Shiite tensions, while Iran-backed Shiite militias are also expected to push for influence.

Mr. Barzani led a high-level Kurdish delegation in meetings this week with top administration officials on continuing the military and political relationship with the Kurdistan Regional Government and Washington.

Despite their cooperation on the battlefield, the relationship between Irbil and Baghdad has been tense. The Kurds claim, among other things, that the national government fails to adequately deliver the funds and resources promised to Iraq’s Kurdish regions.

Tricky political path

Despite continuing fears of domestic unrest in Iraq and in the larger region, the path to a deal on a new status of forces agreement remains politically dangerous.

Shortly after news broke about talks on maintaining American forces in Iraq after the fall of Mosul, Mr. al-Abadi went public with his demand that no U.S. combat troops stay in the country. American military advisers working alongside Iraqi Security Forces would be allowed to remain, ensuring “full readiness” within the country to face any “future security challenges,” The Associated Press reported at the time.

Analysts said his comments reflect the political tightrope Mr. al-Abadi will have to walk with factions in the Iraqi government to complete a troop deal with Washington, rather than the death knell on any status of forces agreement.

“It’s an explosive political issue,” said Jim Phillips, a senior Middle East analyst at The Heritage Foundation, noting Mr. al-Abadi’s public rhetoric may give him space with Masoud Barzani and other power brokers to begin backroom dealing with the Trump administration. Others likely to have a say in the talks include former Prime Minister Nouri al-Maliki, former Iraqi President and Kurdish leader Jalal Talabani and Muqtada al-Sadr, the Shiite cleric, politician and militia leader.

Other personalities who also could have a major impact include Interior Minister Qasim al-Araj and Defense Minister Erfan al-Hiyali.

A former Shiite militia commander, Mr. al-Araji was arrested multiple times by American forces in Iraq on charges of smuggling arms used to attack U.S. troops, before rallying behind U.S. and coalition forces when operations against the Islamic State began in 2015. His powerful ministry controls most of Iraq’s security agencies, including the federal police, and is generally controlled by the pro-Iran Badr Organization.

Mr. al-Hiyali was the commander of the vaunted Iraqi Special Forces unit known as the “Golden Division” before he was appointed as Mr. al-Abadi’s defense chief. The division spearheaded the Iraqi offensive into Islamic State-held Mosul in October and sustained heavy casualties throughout the campaign. His men were also accused of sectarian violence and revenge killings of Islamic State fighters, earning the nickname the “Dirty Brigade.”

As the head of the Iraqi Security Forces, Mr. al-Hiyali also has connections that allow him to speak with Iranian paramilitary organizations tied to the recently federalized Shiite militias known as the Popular Mobilization Units.

Role for Kushner?

Mr. Trump, who has loudly criticized the original mission to go into Iraq in 2003, will have to sell another long-term military engagement to a war-weary public, a job that will likely fall to Mr. Trump’s son-in-law and adviser Jared Kushner.

In April, Mr. Kushner slapped a Kevlar helmet and flak jacket over his chinos and blue blazer on a visit to Baghdad with Joint Chiefs of Staff Chairman Gen. Joseph F. Dunford for a discussion with Iraq’s top military leaders about countering the Islamic State. Mr. Kushner was also one of several top national security officials, including National Security Adviser H.R. McMaster, to meet with Mr. Barzani during his round of White House meetings Tuesday.

The troubled talks between Mr. Obama and Mr. al-Maliki, undertaken as Mr. Obama was promising to end the long U.S. combat missions in Iraq and Afghanistan, show the diplomatic and military minefields both sides will have to navigate.

Obama administration officials wanted legally binding guarantees that U.S. forces based in postwar Iraq would not be subjected to Iraqi jurisdiction. Mr. al-Maliki balked, citing the instances of torture and abuse of Iraqis by U.S. forces at the infamous Abu Ghraib prison and elsewhere across the country. The impasse helped eventually scuttled talks and prompted the 2011 withdrawal.

But Heritage’s Mr. Phillips said Mr. Obama deserved much of the blame for insisting that any status of forces agreement be passed by Iraqi parliamentary vote rather than be issued as an executive agreement by Mr. al-Maliki. That demand helped killed the deal.

“They knew it was a way to get out,” Mr. Phillips said, adding the Obama administration knew Mr. al-Maliki did not have the political capital to push a postwar troop deal through the Iraqi parliament.