Cartagena, Colombia (CNN) — A group of Secret Service agents and officers sent to Colombia ahead of President Barack Obama were relieved of duty and returned home amid allegations of misconduct that involved prostitution, according to two U.S. government sources familiar the investigation.

The 11 Secret Service members — both agents and uniformed officers — were interviewed Saturday at the agency’s Washington headquarters, after which they were placed on administrative leave, Assistant Director Paul Morrissey said in a statement.

They are under investigation after preliminary findings revealed that they brought back several prostitutes to the Hotel Caribe in Cartagena, sources told CNN on Saturday.

Also Saturday, the U.S. military announced that five service members assigned to support the Secret Service in its assignment have been “confined to quarters” in Colombia after they violated curfew and “may have been involved in inappropriate conduct” at the same hotel. The statement, from U.S. Southern Command, did not offer more details, including whether prostitution was involved.

The command’s leader, Gen. Douglas Fraser, said he is “disappointed by the entire incident and that this behavior is not in keeping with the professional standards expected of members of the United States military,” according to the statement.

The alleged misconduct overshadowed the start of the sixth Summit of the Americas, where the president was to focus on trade, energy and regional security.

None of the agents or officers being investigated was part of the president’s personal protective detail and Obama isn’t based at the hotel. But dignitaries and journalists reporting on the hemispheric meeting were staying there, a U.S. government official said.

Obama: ‘Hemisphere is very well positioned in the global economy’

Rep. Peter King, chairman of the House Homeland Security Committee, was briefed on the matter and told CNN that the government personnel brought prostitutes back to their rooms Wednesday night and “one of the women did not leave the room in the morning.”

A hotel manager tried to get in the room, and eventually the woman emerged and said “they owed her money,” according to King. Similarly, U.S. government sources said there was a dispute between at least one Secret Service member and a woman brought back to his hotel over a request to be paid.

At least one of the women brought to the hotel talked with police, and complaints were filed with the U.S. Embassy, the sources said.

“The Secret Service saw that report, and they immediately began an investigation,” King said.

A hotel staffer gave Colombian security personnel the entire list of U.S. government personnel staying there, said a U.S. military official who couldn’t speak for attribution because of the ongoing investigation.

A spokesman for Colombia’s National Police declined to comment, referring questions to the Secret Service.

King, the Republican from New York, said the issue isn’t about any “criminal allegations but a dereliction of duty.”

“My understanding is that there are no allegations of any crime being committed,” he told CNN’s Don Lemon. “It violates the Secret Service code of conduct.”

Secret Service spokesman Edwin Donovon said that the group of agents in Cartegena were relieved of duty Thursday — prior to the president’s arrival in Colombia — and replaced after “allegations of misconduct.”

The U.S. government sources said that, because the trip already had a large security detail, only some of the recalled agents and officers were replaced. Still, the sources said there was no threat to the president.

Morrissey, the assistant director from the Secret Service, noted his agency’s “zero tolerance policy on personal misconduct.” He said that putting the accused employees on leave is “standard procedure” as the agency conducts a “full, thorough and fair investigation.”

“This incident is not reflective of the behavior of our personnel as they travel every day throughout the country and the world performing their duties in a dedicated, professional manner,” Morrissey said. “We regret any distraction from the Summit of the Americas this situation has caused.”

The U.S. military is conducting its own investigation and will mete out “punishment, if appropriate … in accordance with established policies and the Uniform Code of Military Justice,” the Southern Command said in its statement.

A senior military official and senior Secret Service official decided the five U.S. service members would stay in Colombia, confined to their hotel except for work-related business, because their “special skills” were still needed, according to a U.S. military official. Barred from contacting others about the matter, they will return to the United States with the rest of their unidentified unit at the end of the mission, according to the military.

Ronald Kessler, a former Washington Post reporter who has written a book about the Secret Service, called the incident “clearly the biggest scandal in Secret Service history.”

The Washington Post, which was the first to report the story, said it was alerted to the investigation by Kessler.

Jon Adler — president of the Federal Law Enforcement Officers Association, which represents Secret Service agents and other federal law enforcement officers — urged caution in jumping to conclusions, characterizing the incident as “isolated” and not necessarily a scandal.

“That’s just sort of an overdramatic interpretation of an isolated incident,” he said. “We have to trust the process of the internal review.”

While soliciting prostitution is legal in certain areas of Colombia, it is considered a breach of the agency’s conduct code, the government sources said. High-level officials in the Secret Service and the Department of Homeland Security were outraged over the incident, the sources said, noting that the investigation indicated the prostitutes were brought back to a hotel that had been secured for the summit.

The president arrived in Cartagena on Friday, the same day he first learned about the incident, White House press secretary Jay Carney said from the coastal resort city. Obama will spend more time in Colombia, where security concerns had limited previous presidential trips, than any other U.S. president.

Two nearly back-to-back explosions in Cartagena — which caused only minor damage and no casualties, Colombia National Police spokesman Cantihho Toncell said — were a reminder of the violence that has gripped Colombia as its government has battled powerful drug cartels. Violence has significantly fallen off in recent years as the Bogota-based government, aided by U.S. extradition efforts, has successfully picked apart the cartels.

Still, there is an extensive security presence in the walled colonial city of Cartagena for the summit — including more than 7,600 police officers and thousands more troops.