(CNN) - Paul Bremer, former presidential envoy to Iraq, said Monday that there will have to be special forces on the ground to fight the insurgency from militant groups in Iraq.

In a tense exchange with CNN's Erin Burnett on "OutFront," Bremer also blamed President Barack Obama for pulling out all forces in 2011 instead of leaving a residual troop presence to ensure security in the region.

"We need to get away from this idea that we're not going to have troops on the ground but we will have, we will have to have special forces and intelligence observers, fire control officers, people identifying targets in those cities so that drones can hit them. You can't subdue a city with fighter jets, you're going to need drones," he said.

Bremer's comments come as U.S. troops head to Iraq following the Islamic State in Iraq and Syria's offensive that has seen vast swaths of northern Iraq fall out of government hands.

"After 4,490 lives have been lost in the Iraq war. And $1.7 trillion spent already. How can you advocate any more people, any more lives going to risk for that country?" Burnett said.

Bremer responded: "Because it's in our interest."

"We cannot allow the world's worst terrorist group to get a base of support in a failed nation," he added, warning that instability in Iraq could spell trouble for allies in the region.

Burnett later asked: "What I don't understand is how you can say it was a success after President George W. Bush left office, President Obama was put in a position where he had to do a second surge because Iraq was not stable?"

"Our departure in 2011 was a signal to the Iraqis that we were leaving, we were out and two things then happened - we lost our ability to work closely with the Iraqis on intelligence and training and we lost our political influence," Bremer said,

Bremer called the withdraw of troops a "serious mistake" by Obama and argued that the absence of a U.S. presence put Iraqi Prime Minister Nuri al-Maliki in an "impossible situation" to control militant forces.