President Obama said today that he expects a "thorough" and "rigorous" investigation of allegations of misconduct involving members of the Secret Service and military assigned to a security detail ahead of his visit to Cartagena, Colombia.

"If it turns out that some of the allegations that have been made in the press are confirmed, then of course I'll be angry," Obama said during a press conference with Colombian President Juan Manuel Santos following a bi-lateral meeting.

The comments were the first public statement by the president on the scandal, which has cast a shadow over his trip to the Summit of the Americas.

Obama said he holds the Secret Service to the same high standards as the rest of his staff, expecting them to conduct themselves with "dignity and probity."

"We're representing the people of the United States, and when we travel to another country I expect us to observe the highest standards because we're not just representing ourselves," he said.

While Obama and the White House have praised the "extraordinary service" of the agents and sought to downplay the scandal's significance, Secret Service and military officials and members of Congress have called the alleged behavior outrageous and launched a methodical fact-finding investigation.

Since adult prostitution is legal in Colombia in designated "tolerance zones," officials said the investigation would center less on moral or legal aspects of the alleged behavior and more on whether Secret Service and military protocols were violated -- and whether the security of the president could have been compromised.

"If all this happened, this compromised the agents themselves," House Homeland Security Committee chairman Rep. Peter King, R-N.Y., told ABC News. "It left [the agents] open to be threatened and blackmailed in the future. ... They could have been threatened or blackmailed secondly to bring prostitutes in an area that's a secured zone. It just violates a basic code of conduct."

Rep. Darrell Issa, R-Calif., who chairs the House Oversight and Government Affairs Committee, said Congress will also scrutinize the Secret Service as a whole to determine whether and how often similar situations may have happened before.

"In this particular case, the president may not have been in danger. But that begs the question -- what happens if somebody six months ago, six years ago became the victim of their own misconduct and is now being blackmailed?" Issa said on CBS' "Face the Nation."

"The question is, is the whole organization in need of some soul searching, some changes before the president, the vice president, members of the cabinet are in danger?" Issa said.

The Secret Service members were interviewed Saturday in Washington and have been placed on administrative leave. If the allegations are proven true, they could face reprimands and could be fired.

The Defense Department restricted to their quarters five personnel who were assigned to assist the Secret Service for alleged participation in the inappropriate conduct. They will return to the United States for questioning at the conclusion of the mission, officials said.

The House Homeland Security Committee, which oversees the Secret Service, has also launched an inquiry.

A heated argument between at least one of the alleged prostitutes and at least one of the Secret Service agents on Thursday first alerted local authorities to the alleged misbehavior, officials told ABC News.