Alex Wong/Getty

For years, CIA Director John Brennan and White House Chief of Staff Denis McDonough were part of a small group of Obama administration officials who believed that five relatively high-ranking Taliban commanders could be released under certain conditions with manageable risk of harm to American interests. For years, senior defense and intelligence officials disagreed—and were poised to block a potential trade for the Taliban five and American hostage Bowe Bergdahl.

By 2014, many of the skeptics had left the Obama administration; Brennan, McDonough, and their allies assumed new roles at the very top of the Obama administration; and the White House and its allies at State were able to convince their replacements to sign off on the deal.

“All of us on the National Security team were unanimous in supporting and recommending that we take this opportunity,” National Security Advisor Susan Rice told CNN Friday. But for years, that was not the case and Brennan and McDonough were opposed to other senior officials.

Inside the administration, some officials working on the swap of the five Taliban commanders for captured U.S. soldier Bowe Bergdahl did not see the prisoners as the “worst of the worst,” as many GOP senators have said this week. As far back as 2009, when the original idea of a swap was floated, some officials held the position these five prisoners were essentially Afghans dedicated to fighting other Afghans, not international terrorists.

“Look, these were not good guys. I am in no way defending these men,” State Department Spokeswoman Marie Harf told reporters Thursday. “But being mid- to high-level officials in a regime that’s grotesque and horrific also doesn’t mean they themselves directly pose a threat to the United States.”

In late 2011 and early 2012, administration officials who thought the risk of releasing the Taliban five was manageable included Brennan—then the White House counterterrorism czar—State Department official Marc Grossman, and then-Deputy National Security Advisor Denis McDonough, who led the interagency process that ultimately decided to make an initial prisoner swap offer to the Taliban in the fall of 2011.

The deal proposed to the Taliban in 2011 was very different than the final deal struck in 2014 and not all of the details of the old deal have been publicly revealed. For example, the 2011 deal would have only released 3 Taliban prisoners first, then after 60 days, 2 more. The old deal also included demands that the Taliban take steps towards a peace and reconciliation process with the Afghan government. The prisoner swap was only one part of a larger package.

McDonough first became close to the President in 2007, when he was still a Senator. And Brennan’s voice carried particular weight. A former CIA station chief in Saudi Arabia, Brennan in 2009 became Obama's tutor on the classified side of the war on terror and his envoy to the spies, special operators and analysts fighting that war. Brennan, McDonough, and the others were in favor of offering a prisoner swap to the Taliban only if they agreed to strict conditions on monitoring in Qatar and as part of a sequence of events aimed at getting the Taliban into a reconciliation process.

(The CIA declined to comment for this story. The White House did not respond to a request for comment.)

After a vigorous internal debate, the White House decided to offer the Taliban the deal in 2011, But the terms and circumstances were different than the agreement later struck between the U.S. and the Taliban in 2014.

Back in 2011, the idea these five prisoners could be released safely was opposed by Leon Panetta, who served in Obama’s first term as secretary of defense and director of the CIA. James Clapper, the director of national intelligence, was also opposed. But Brennan and others argued that the Taliban five were primarily focused on fighting against other Afghans and never had a record of attacking Americans outside of their own country. They had extensive ties to al Qaeda, but were focused on their own civil war, not international jihad.

“Yes they had ties to al Qaeda, but there’s no evidence at all that any of them knew of or supported 9-11 or did anything to support al Qaeda’s international agenda,” a former Obama administration official who supported that view told The Daily Beast.

All five Taliban leaders were captured in the weeks just after America attacked Afghanistan, and according to the Wikileaked classified intelligence dosiers compiled by their Guantanamo jailers, at least two of them were trying to surrender when they were detained and shipped off to Cuba. Another one may have been working with the CIA. None were in Afghanistan during the bulk of the fighting between the Taliban and U.S. forces.

“None of them actually participated in the resistance,” this official said.

Nonetheless, these arguments for years fell on deaf ears in the intelligence community. Speaking in Pittsburgh this week Panetta said of the trade for Bergdahl: “I don't fault the administration for wanting to get him back. I do question whether the conditions are in place to make sure these terrorists don't go back into battle.”

Also, as recently as 2013, the U.S. military’s assessment of the risks posed by the Taliban five if released still assessed it was likely they would return to the fight against Americans, at least in Afghanistan.

There are some indications these warnings might someday be proven true. NBC reported Friday that one of the detainees, Noorullah Noori pledged to return to Afghanistan to kill Americans, according to a second-hand account from an unnamed Taliban commander who claimed to have spoken with him.

Those inside the administration who supported releasing the prisoners noted that the Afghan government and President Hamid Karzai were lobbying for their release. Karzai did interfere with the negotiations at several points, in order to protect his own interests. But he repeatedly told the Obama administration that the Taliban five should be released back to Afghanistan with no restrictions at all, according to multiple senior U.S. and Afghan officials.

“The Taliban wanted them released to Afghanistan. Karzai also wanted us to release them to Afghanistan. He objected to the transfer because of all the conditions. He said this was Afghans being transferred from one prison to another prison,” the former Obama administration official said.

The secret dossiers on the Taliban five prepared by Joint Task Force, Guantanamo Bay, judge that all five are a “high risk” to return to the fight in Afghanistan, the long-held military judgment. But they also detail the unique circumstances of the capture and transfer of these five men to Cuba.

Mohammad A Fazl, former Chief of Staff of the Taliban Army and his deputy Noori were almost surely responsible for the slaughter of thousands of Shiite Muslims in Northern Afghanistan during the Taliban’s war with the Northern Alliance. They fought against U.S. and coalition forces who attacked in October, 2001, but then attempted to surrender to a Northern Alliance Commander General Dostum with many of their men only a few weeks later, according to the U.S. military documents.

“Taliban military commanders were informed if they surrendered their weapons to General Dostum, they would be allowed to return home,” the document said.

The Taliban even paid Dostum $500,000 to secure Fazl’s safe passage after his surrender. But following Fazl and Noori’s refusal to help quell a Taliban prisoner resistance at the facility were Dostum was holding them, in December 2001, Dostum handed over Fazl and Noori to U.S. forces, who sent them to Guantanamo.

Fazl and Noori both had extensive connections to al Qaeda, according to the U.S. military documents.

Khair Ulla Said Wali Khairkhwa was a notorious heroin trafficker. He was also at different times The Taliban Interior Minister, the official Taliban spokesman and the governor of Herat Province under Taliban rule. He also may have tried to defect from the Taliban after they lost the war and join up with the Afghan government. He claimed he was in the process of switching sides when he was arrested and sent to Guantanamo.

“Detainee is a friend of current Afghan President, Hamid Karzai,” his secret military file shows. “When the Taliban lost control, detainee contacted Karzai to discuss a position with the new government and detainee’s personal safety. Several Karzai associates met with detainee in the time between the Taliban’s fall and his arrest.”

Kairkhwa also tried to broker a peace in 2001 between the Taliban and the Northern Alliance with Iranian help, but that failed. Because of his ties to all sides, Karzai’s government believes he can be a helpful in achieving Afghan reconciliation.

Mohammed Nabi Omari is the only member of the Taliban five with strong connections to the Haqqani network, the Pakistan-based military group that actually held and guarded Bergdahl for his five years of imprisonment. Just prior to his arrest, he met several times with a CIA operative named “Mark,” who gave him money and tasked him to find the location of Taliban leader Mullah Omar. Omari didn’t produce Mullah Omar’s location and according to him, upon arriving at a scheduled meeting with “Mark,” he was detained and sent to Cuba. He also had extensive al Qaeda ties.

Abdul Haq Wasiq was the former Taliban deputy minister of intelligence and had also been in contact with U.S. intelligence officials claiming to want to help them fund Mullah Omar. He requested a GPS device from the Americans and radio frequencies to transmit information back to the U.S. forces. He was arrested shortly thereafter. His file shows that he was crucial in joining Taliban and al Qaeda intelligence capabilities in the years before the U.S. invasion.

By 2014, when the Bowe Bergdahl prisoner swap deal was finally struck, the circumstances surrounding the deal had completely changed. Brennan was in charge of the CIA and McDonough was the chief of staff at a White House now firmly in control of the deal-making process. Obama’s team was planning to announce the end of U.S. combat operations in 2014 and the complete withdrawal of U.S. troops by 2017.

The prisoner swap was no longer part of a sequence toward a peace process; that would now be the Afghan government’s responsibility alone. And without U.S. forces in combat, some inside the administration believed the legal justification for holding Taliban leaders as “enemy combatants” was sure to come into question.

“When 2014 ends, the White House will have to formulate a legal doctrine under which we can detain law of war detainees. They have not yet formulated that document,” the former administration official said.

The struggle over who to release from Guantanamo will rage on inside the administration between those defense and military leaders who want to keep the prisoners there and those in the White House and State Department who want to see the Guantanamo prison closed.

As for the military and intelligence officials who warned that the Taliban five were a “high risk” to Americans if they were released, the official said: “We’ll find out now that they are released if that was overstated it or not.”