What a hangover. Accusations of misconduct among U.S. security personnel in Colombia keep flooding in, casting a shadow over President Obama’s trip to the Summit of the Americas conference in Cartagena and embarrassing a proud U.S. agency. The allegations were even strong enough to garner a response from Obama himself.

The saga began when reports emerged Friday evening that around a dozen Secret Service members were relieved of duty in Colombia after allegedly taking prostitutes to Hotel Caribe in Cartagena, where they were staying. The agents had been sent to Colombia ahead of President Obama’s arrival and were said to have been sent home on Thursday, just a day before the President landed for the summit, which is hosting leaders from 33 countries in the western hemisphere. According to the Christian Science Monitor, the agents had been partying and drinking heavily during their stay. Saturday evening, the Secret Service confirmed that 11 agents had been placed on leave for their misconduct.

Representative Peter King, a New York Republican, told the Associated Press that the scandal involved 11 agents. None were members of the President’s personal security detail, and the hotel they occupied is not the one where Obama is now staying. King explained that some of the agents are accused of inviting women to stay the night in their hotel and that some of those women were “presumed to be prostitutes,” though it’s unclear if the agents knew that at the time. While prostitution is legal in “tolerance zones” in certain Colombian cities, it’s widely overlooked by authorities — and certainly in violation of Secret Service rules.

The New York Times reports that one agent was said to have had a dispute with a woman over money and forced her to leave his room, causing a disturbance that attracted other agents in adjacent rooms and eventually brought out hotel security and Colombian police. “There are people who willingly went to prostitutes and other people who ended up with prostitutes,” an unnamed official told the New York Times. “Either way, it’s just unacceptable.”

The embarrassing news traveled back to the U.S. with the accused agents. And now officials are wondering how the scandal will affect the reputation of the Secret Service, whose members are known for their immaculate black suits, sunglasses, earpieces and, most important, their professionalism. The agency’s motto is “Worthy of Trust and Confidence.”

White House spokesman Jay Carney on Saturday said that the allegations were not a distraction for the U.S. delegation, and that the President maintains “full confidence” in the Secret Service. Carney noted that it “would not be appropriate” for President Obama to comment on the situation.

This made Obama’s response Sunday evening all the more surprising. The President commented on the accusations Sunday as media reports continued to cloud the summit’s primary mission. Without noting the specific details of the scandal, he reprimanded the 11 agents allegedly involved: “If it turns out that some of the allegations that have been made in the press are confirmed, then of course I’ll be angry.”

He responded to a question about the scandal during a press conference at the summit’s closing session, flanked by Colombian President Juan Manuel Santos. “I expect that investigation to be thorough, and I expect it to be rigorous,” Obama said, while noting that the allegations haven’t bucked his confidence in the Secret Service. He explained that he remains “very grateful” for the “outstanding job” done by the agents who are responsible for his safety.

In a statement, a Secret Service spokesman declined to go into specifics but made it clear that the alleged wrongdoers were no longer on the job in Colombia. “Those personnel are being relieved of their assignments, returned to their place of duty and are being replaced by other Secret Service personnel,” said agency spokesman Edwin M. Donovan. He wouldn’t confirm the exact number of team members involved, but noted that the investigation has been turned over to the Office of Professional Responsibility, the Secret Service’s internal-affairs team. Representative Darrell Issa, who leads the House Oversight and Government Reform Committee, expects the investigation to have wider-reaching consequences. “Things like this don’t happen once if they didn’t happen before,” he said Sunday on CBS’s Face the Nation.

On Saturday, Politico reported that five members of the U.S. military’s Southern Command stationed in Latin America and on hand to assist the Secret Service team in Colombia might have also been involved in allegedly taking prostitutes to the hotel. Southern Command commander General Douglas Fraser said he is “disappointed by the entire incident and that this behavior is not in keeping with the professional standards expected of members of the United States military.” The five service members in question have been confined to quarters as the military begins its own investigation.

But the scandal has no doubt shifted international focus off of the diplomatic mission that took Obama to Latin America. Economic cooperation was one of the issues on his agenda, but not the kind that ended up in Saturday’s papers.