BAGHDAD—Iraq's president appointed a candidate to replace Prime Minister Nouri al-Maliki but the longtime leader refused to step aside, setting up a confrontation as the government struggles to combat a rapidly advancing Sunni insurgency.

President Fuad Masum designated Haider al-Abadi, a member of Mr. Maliki's Shiite Islamist Dawa party, to form the next government. Mr. Maliki, who was vying for a third four-year term, angrily...