Washington (CNN) -- Three of the 12 Secret Service agents involved in the Colombia prostitution scandal refused to cooperate with authorities and submit to a polygraph test, according to House Homeland Security Committee Chairman Peter King.

The three agents were among the first forced out of the service when news of the scandal in Cartagena broke, King told CNN late Tuesday. The nine remaining agents took the polygraph. And while none of them failed the test, some responses led to the loss of several jobs.

King, R-New York, received the information as a part of a response to 50 questions he sent to Secret Service Director Mark Sullivan on April 20. Sullivan submitted his answers to King Tuesday.

In an interview Wednesday morning on CNN, King said no security breach occurred from the Secret Service agents consorting with foreign prostitutes in their hotel rooms shortly before President Barack Obama's arrival in Cartagena, Colombia, for last month's Summit of the Americas.

"We know that it appear that no material was obtained by any of the prostitutes, nothing is missing, all the BlackBerrys are accounted for, there was no president's schedule available," King told CNN's "Starting Point with Soledad O'Brien."

"It does not appear that any of the 12 women had any involvement other than prostitution," King added. "They were not working for any narco-terrorist organization, and I think in a way the Secret Service has ducked a bullet."

Still, King said, the incident "goes against all the principles of the Secret Service."

"Because it was disclosed and there was no long-term security matter here, it gives the Secret Service the opportunity to clear up what has happened, do all it can to make sure it never happens again or at least minimizes it to make it very, very difficult for it to ever happen again," he said.

While King did not provide CNN copies of the responses -- which he said are marked "law enforcement sensitive" -- he highlighted several details Tuesday night.

Among other things, one agent said in the polygraph test that he was "actively engaged" with one of the prostitutes when he said she wanted to get paid, King said. In response, the agent threw her out of his room.

The agent told U.S. officials he didn't realize the woman was a prostitute, and has not been fired.

U.S. officials have now interviewed 10 of the 12 women involved in the scandal, King noted. The Secret Service and Colombian authorities are trying to track down the remaining two.

King stressed what he called a "pleasant surprise" -- Sullivan's decision to call the Homeland Security Department's inspector general before bringing the agents back from Colombia.

King said there weren't many surprises in the responses to his questionnaire.

"Sullivan was giving us good information all along," he told CNN.

House Oversight Committee Chairman Darrell Issa and Rep. Elijah Cummings, the panel's ranking Democrat, have submitted another 10 questions to Sullivan, including asking for a precise time line of exactly what happened in Cartagena.

The pair also sent a letter to Defense Secretary Leon Panetta requesting details of a military investigation by May 8.

In their correspondence to Panetta, Issa and Cummings said security personnel showed an "alarming lack" of "character" and "judgment."

The Homeland Security Department's inspector general is currently investigating the scandal, as are four congressional committees. The agency, the military and the White House are also conducting internal reviews of the incident.

The top legislators on the Senate Homeland Security and Governmental Affairs Committee said earlier Tuesday they've also sent a letter to Sullivan asking for information on the incident. A total of nine agents have resigned or are in the process of being forced out.

Three other Secret Service agents were cleared of serious misconduct, and the military is investigating the alleged involvement of 12 of its service members.

On Monday, the Homeland Security official announced his separate investigation of the incident, which embarrassed the government and raised questions of a possible security breach.

The "field work is beginning immediately," acting Inspector General Charles Edwards said in a statement.

The Senate Judiciary Committee questioned Homeland Security Secretary Janet Napolitano about the controversy at a hearing last week. On Tuesday, Sen. Joe Lieberman, an independent from Connecticut who chairs the Senate Homeland Security Committee, and ranking Republican Sen. Susan Collins said they sent Sullivan a letter Monday that also sought answers about what happened.

"We wish to determine whether those events were indicative of a pattern of behavior by agents or officers of the Secret Service, and need to be addressed systemically, or if they instead constituted an isolated incident warranting action only with respect to the individuals involved," said the letter from Lieberman and Collins.

The U.S. Southern Command expects to finish questioning the 12 military personnel implicated in possible wrongdoing this week before forwarding its findings to military lawyers for review, and then to Gen. Douglas Fraser, commanding general of the U.S. Southern Command, a Defense Department official said Monday.

Last week, the Secret Service distributed new rules for its agents on assignment intended to prevent a repeat of such alleged misconduct, according to two government sources familiar with the resulting investigation.

Enhanced Standards of Conduct, the new guidelines given to all Secret Service personnel, make clear that standards of behavior required in the United States apply on missions abroad, the sources said.

Effective immediately, the new standards require detailed briefings before each trip that will include safety precautions and any necessary designations of establishments and areas that are "off limits" for Secret Service personnel, the sources said.

Also in the new standards, foreigners are banned from Secret Service hotel rooms at all times, except for hotel staff and host nation law enforcement and government officials on official business, according to the officials, and all Secret Service personnel are prohibited from going to a "nonreputable establishment."

The new standards specify that U.S. laws apply to Secret Service personnel when traveling, rendering invalid the excuse that specific activity is legal in the foreign country, the officials said.

In addition, the new guidelines allow moderate alcohol consumption when off duty, but prohibit alcohol consumption within 10 hours of reporting for duty or at any time when at the hotel where the protected official is staying, the officials explained.

An additional supervisor from the Office of Professional Responsibility will now accompany the "jump teams" that bring vehicles for motorcades and other transportation, the officials said. Agents involved in the Colombia incident were part of such a jump team.

Allegations of further transgressions by agents have emerged after the initial reports of heavy drinking and consorting with prostitutes last month before Obama arrived in Cartagena.

Recent claims include an account from El Salvador described by CNN affiliate Seatte TV station KIRO as very similar to the Colombia scandal, involving members of the Secret Service and other government agencies.

However, Panetta said last week that his department is not investigating any of its troops over the reported incident in El Salvador, while State Department spokesman Victoria Nuland said embassy staff in El Salvador were being questioned about the allegations.

The Drug Enforcement Administration also is prepared to look into, "in an appropriate manner and immediately," allegations that it deems "credible" regarding its agents in El Salvador, agency spokesman Rusty Payne said. But he added that, while the DEA had seen news reports, "we are unaware of any allegations of misconduct."

Online Public Information File

Viewers with disabilities can get assistance accessing this station's FCC Public Inspection File by contacting the station with the information listed below. Questions or concerns relating to the accessibility of the FCC's online public file system should be directed to the FCC at 888-225-5322, 888-835-5322 (TTY), or fccinfo@fcc.gov.