In 2004, with the war in Iraq going from bad to worse, the US drafted in a veteran of Central America's dirty wars to help set up a new force to fight the insurgency. The result: secret detention centres, torture and a spiral into sectarian carnage

An exclusive golf course backs onto a spacious two-storey house. A coiled green garden hose lies on the lawn. The grey-slatted wooden shutters are closed. And, like the other deserted luxury houses in this gated community near Bryan, Texas, nothing moves.

Retired Colonel Jim Steele, whose military decorations include the Silver Star, the Defence Distinguished Service Medal, four Legions of Merit, three Bronze Stars and the Purple Heart, is not at home. Nor is he at his office headquarters in Geneva, where he is listed as the chief executive officer of Buchanan Renewables, an energy company. Similar efforts to track him down at his company's office in Monrovia are futile. Messages are left. He doesn't call back.

For over a year the Guardian has been trying to contact Steele, 68, to ask him about his role during the Iraq war as US defence secretary Donald Rumsfeld's personal envoy to Iraq's Special Police Commandos: a fearsome paramilitary force that ran a secret network of detention centres across the country – where those suspected of rebelling against the US-led invasion were tortured for information.

On the 10th anniversary of the Iraq invasion the allegations of American links to the units that eventually accelerated Iraq's descent into civil war cast the US occupation in a new and even more controversial light. The investigation was sparked over a year ago by millions of classified US military documents dumped onto the internet and their mysterious references to US soldiers ordered to ignore torture. Private Bradley Manning, 25, is facing a 20-year sentence, accused of leaking military secrets.

Steele's contribution was pivotal. He was the covert US figure behind the intelligence gathering of the new commando units. The aim: to halt a nascent Sunni insurgency in its tracks by extracting information from detainees.

It was a role made for Steele. The veteran had made his name in El Salvador almost 20 years earlier as head of a US group of special forces advisers who were training and funding the Salvadoran military to fight the FNLM guerrilla insurgency. These government units developed a fearsome international reputation for their death squad activities. Steele's own biography describes his work there as the "training of the best counterinsurgency force" in El Salvador.

Of his El Salvador experience in 1986, Steele told Dr Max Manwaring, the author of El Salvador at War: An Oral History: "When I arrived here there was a tendency to focus on technical indicators … but in an insurgency the focus has to be on human aspects. That means getting people to talk to you."

But the arming of one side of the conflict by the US hastened the country's descent into a civil war in which 75,000 people died and 1 million out of a population of 6 million became refugees.

Celerino Castillo, a Senior Drug Enforcement Administration special agent who worked alongside Steele in El Salvador, says: "I first heard about Colonel James Steele going to Iraq and I said they're going to implement what is known as the Salvadoran Option in Iraq and that's exactly what happened. And I was devastated because I knew the atrocities that were going to occur in Iraq which we knew had occurred in El Salvador."

It was in El Salvador that Steele first came in to close contact with the man who would eventually command US operations in Iraq: David Petraeus. Then a young major, Petraeus visited El Salvador in 1986 and reportedly even stayed with Steele at his house.

But while Petraeus headed for the top, Steele's career hit an unexpected buffer when he was embroiled in the Iran-Contra affair. A helicopter pilot, who also had a licence to fly jets, he ran the airport from where the American advisers illegally ran guns to right-wing Contra guerrillas in Nicaragua. While the congressional inquiry that followed put an end to Steele's military ambitions, it won him the admiration of then congressman Dick Cheney who sat on the committee and admired Steele's efforts fighting leftists in both Nicaragua and El Salvador.

In late 1989 Cheney was in charge of the US invasion of Panama to overthrow their once favoured son, General Manuel Noriega. Cheney picked Steele to take charge of organising a new police force in Panama and be the chief liaison between the new government and the US military.

Todd Greentree, who worked in the US embassy in El Salvador and knew Steele, was not surprised at the way he resurfaced in other conflict zones. "It's not called 'dirty war' for nothing; so it's no surprise to see individuals who are associated and sort of know the ins-and-outs of that kind of war, reappear at different points in these conflicts," he says.

A generation later, and half the world away, America's war in Iraq was going from bad to worse. It was 2004 – the neo-cons had dismantled the Ba'athist party apparatus, and that had fostered anarchy. A mainly Sunni uprising was gaining ground and causing major problems in Fallujah and Mosul. There was a violent backlash against the US occupation that was claiming over 50 American lives a month by 2004.

The US Army was facing an unconventional, guerrilla insurgency in a country it knew little about. There was already talk in Washington DC of using the Salvador option in Iraq and the man who would spearhead that strategy was already in place.

Soon after the invasion in March 2003 Jim Steele was in Baghdad as one of the White House's most important "consultants", sending back reports to Rumsfeld. His memos were so valued that Rumsfeld passed them on to George Bush and Cheney. Rumsfeld spoke of him in glowing terms. "We had discussion with General Petraeus yesterday and I had a briefing today from a man named Steele who's been out there working with the security forces and been doing a wonderful job as a civilian as a matter of fact."

In June 2004 Petraeus arrived in Baghdad with the brief to train a new Iraqi police force with an emphasis on counterinsurgency. Steele and serving US colonel James Coffman introduced Petraeus to a small hardened group of police commandos, many of them among the toughest survivors of the old regime, including General Adnan Thabit, sentenced to death for a failed plot against Saddam but saved by the US invasion. Thabit, selected by the Americans to run the Special Police Commandos, developed a close relationship with the new advisers. "They became my friends. My advisers, James Steele and Colonel Coffman, were all from special forces, so I benefited from their experience … but the main person I used to contact was David Petraeus."

With Steele and Coffman as his point men, Petraeus began pouring money from a multimillion dollar fund into what would become the Special Police Commandos. According to the US Government Accounts Office, they received a share of an $8.2bn (£5.4bn) fund paid for by the US taxpayer. The exact amount they received is classified.

With Petraeus's almost unlimited access to money and weapons, and Steele's field expertise in counterinsurgency the stage was set for the commandos to emerge as a terrifying force. One more element would complete the picture. The US had barred members of the violent Shia militias like the Badr Brigade and the Mahdi Army from joining the security forces, but by the summer of 2004 they had lifted the ban.

Shia militia members from all over the country arrived in Baghdad "by the lorry-load" to join the new commandos. These men were eager to fight the Sunnis: many sought revenge for decades of Sunni-supported, brutal Saddam rule, and a chance to hit back at the violent insurgents and the indiscriminate terror of al-Qaida.

Petraeus and Steele would unleash this local force on the Sunni population as well as the insurgents and their supporters and anyone else who was unlucky enough to get in the way. It was classic counterinsurgency. It was also letting a lethal, sectarian genie out of the bottle. The consequences for Iraqi society would be catastrophic. At the height of the civil war two years later 3,000 bodies a month were turning up on the streets of Iraq — many of them innocent civilians of sectarian war.

But it was the actions of the commandos inside the detention centres that raises the most troubling questions for their American masters. Desperate for information, the commandos set up a network of secret detention centres where insurgents could be brought and information extracted from them.

The commandos used the most brutal methods to make detainees talk. There is no evidence that Steele or Coffman took part in these torture sessions, but General Muntadher al Samari, a former general in the Iraqi army, who worked after the invasion with the US to rebuild the police force, claims that they knew exactly what was going on and were supplying the commandos with lists of people they wanted brought in. He says he tried to stop the torture, but failed and fled the country.

"We were having lunch. Col Steele, Col Coffman, and the door opened and Captain Jabr was there torturing a prisoner. He [the victim] was hanging upside down and Steele got up and just closed the door, he didn't say anything – it was just normal for him."

He says there were 13 to 14 secret prisons in Baghdad under the control of the interior ministry and used by the Special Police Commandos. He alleges that Steele and Coffman had access to all these prisons and that he visited one in Baghdad with both men.

"They were secret, never declared. But the American top brass and the Iraqi leadership knew all about these prisons. The things that went on there: drilling, murder, torture. The ugliest sort of torture I've ever seen."

According to one soldier with the 69th Armoured Regiment who was deployed in Samarra in 2005 but who doesn't want to be identified: "It was like the Nazis … like the Gestapo basically. They [the commandos] would essentially torture anybody that they had good reason to suspect, knew something, or was part of the insurgency … or supporting it, and people knew about that."

The Guardian interviewed six torture victims as part of this investigation. One, a man who says he was held for 20 days, said: "There was no sleep. From the sunset, the torture would start on me and on the other prisoners.

"They wanted confessions. They'd say: 'Confess to what have you done.' When you say: 'I have done nothing. Shall I confess about something I have not done?', they said: 'Yes, this is our way. The Americans told us to bring as many detainees as possible in order to keep them frightened.'

"I did not confess about anything, although I was tortured and [they] took off my toenails."

Neil Smith, a 20-year-old medic who was based in Samarra, remembers what low ranking US soldiers in the canteen said. "What was pretty widely known in our battalion, definitely in our platoon, was that they were pretty violent with their interrogations. That they would beat people, shock them with electrical shock, stab them, I don't know what else ... it sounds like pretty awful things. If you sent a guy there he was going to get tortured and perhaps raped or whatever, humiliated and brutalised by the special commandos in order for them to get whatever information they wanted."

He now lives in Detroit and is a born-again Christian. He spoke to the Guardian because he said he now considered it his religious duty to speak out about what he saw. "I don't think folks back home in America had any idea what American soldiers were involved in over there, the torture and all kinds of stuff."

Through Facebook, Twitter and social media the Guardian managed to make contact with three soldiers who confirmed they were handing over detainees to be tortured by the special commandos, but none except Smith were prepared to go on camera.

"If somebody gets arrested and we hand them over to MoI they're going to get their balls hooked, electrocuted or they're going to get beaten or raped up the ass with a coke bottle or something like that," one said.

He left the army in September 2006. Now 28, he works with refugees from the Arab world in Detroit teaching recent arrivals, including Iraqis, English.

"I suppose it is my way of saying sorry," he said.

When the Guardian/BBC Arabic posed questions to Petraeus about torture and his relationship with Steele it received in reply a statement from an official close to the general saying, "General (Ret) Petraeus's record, which includes instructions to his own soldiers … reflects his clear opposition to any form of torture."

"Colonel (Ret) Steele was one of thousands of advisers to Iraqi units, working in the area of the Iraqi police. There was no set frequency for Colonel Steele's meetings with General Petraeus, although General Petraeus did see him on a number of occasions during the establishment and initial deployments of the special police, in which Colonel Steele played a significant role."

But Peter Maass, then reporting for the New York Times, and who has interviewed both men, remembers the relationship differently: "I talked to both of them about each other and it was very clear that they were very close to each other in terms of their command relationship and also in terms of their ideas and ideology of what needed to be done. Everybody knew that he was Petraeus's man. Even Steele defined himself as Petraeus's man."

Maass and photographer Gilles Peress gained a unique audience with Steele at a library-turned-detention-centre in Samarra. "What I heard is prisoners screaming all night long," Peress said. "You know at which point you had a young US captain telling his soldiers, don't, just don't come near this."

Two men from Samarra who were imprisoned at the library spoke to the Guardian investigation team. "We'd be tied to a spit or we'd be hung from the ceiling by our hands and our shoulders would be dislocated," one told us. The second said: "They electrocuted me. They hung me up from the ceiling. They were pulling at my ears with pliers, stamping on my head, asking me about my wife, saying they would bring her here."

According to Maass in an interview for the investigation: "The interrogation centre was the only place in the mini green zone in Samarra that I was not allowed to visit. However, one day, Jim Steele said to me, 'hey, they've just captured a Saudi jihadi. Would you like to interview him?'

"I'm taken not into the main area, the kind of main hall – although out the corner of my eye I can see that there were a lot of prisoners in there with their hands tied behind their backs – I was taken to a side office where the Saudi was brought in, and there was actually blood dripping down the side of this desk in the office.

Peress picks up the story: "We were in a room in the library interviewing Steele and I look around and I see blood everywhere, you know. He (Steele) hears the scream from the other guy who's being tortured as we speak, there's the blood stains in the corner of the desk in front of him."

Maass says: "And while this interview was going on with this Saudi with Jim Steele also in the room, there were these terrible screams, somebody shouting Allah Allah Allah. But it wasn't kind of religious ecstasy or something like that, these were screams of pain and terror."

One of the torture survivors remembers how Adnan Thabit "came into the library and he told Captain Dorade and Captain Ali, go easy on the prisoners. Don't dislocate their shoulders. This was because people were having to undergo surgery when they were released from the library."

General Muntadher fled after two close colleagues were killed after they were summoned to the ministry, their bodies found on a rubbish tip. He got out of Iraq and went to Jordan. In less than a month, he says, Steele contacted him. Steele was anxious to meet and suggested he come to the luxury Sheraton hotel in Amman where Steele was staying. They met in the lobby at 8pm and Steele kept him talking for nearly two hours.

"He was asking me about the prisons. I was surprised by the questions and I reminded him that these were the same prisons where we both used to work. I reminded him of the incident where he had opened the door and Colonel Jabr was torturing one of the prisoners and how he didn't do anything. Steele said: 'But I remember that I told the officer off'. So I said to him: 'No, you didn't — you didn't tell the officer off. You didn't even tell General Adnan Thabit that this officer was committing human rights abuses against these prisoners'. And he was silent. He didn't comment or answer. I was surprised by this."

According to General Muntadher: "He wanted to know specifically: did I have any information about him, James Steele? Did I have evidence against him? Photographs, documents: things which proved he committed things in Iraq; things he was worried I might reveal. This was the purpose of his visit.

"I am prepared to go to the international court and stand in front of them and swear that high-ranking officials such as James Steele witnessed crimes against human rights in Iraq. They didn't stop it happening and they didn't punish the perpetrators."

Steele, the man, remains an enigma. He left Iraq in September 2005 and has since pursued energy interests, joining the group of companies of Texas oilman Robert Mosbacher. Until now he has stayed where he likes to be – far from the media spotlight. Were it not for Bradley Manning's leaking of millions of US military logs to Wikileaks, which lifted the lid on alleged abuses by the US in Iraq, there he may well have remained. Footage and images of him are rare. One video clip just 12 seconds long features in the hour-long TV investigation into his work. It captures Steele, then a 58-year-old veteran in Iraq, hesitating, looking uncomfortable when he spots a passing camera.

He draws back from the lens, watching warily out of the side of his eye and then pulls himself out of sight.