The Pentagon has suspended security clearances of military personnel implicated in the Colombia prostitution scandal, pending the results of the investigation, Defence Secretary Leon Panetta has said.

US Secret Service and military personnel allegedly took as many as 21 women back to their hotel, ahead of President Barack Obama's trip to the seaside city of Cartagena to attend the Summit of the Americas.

They were discovered when one woman complained about money, leading to the involvement of the local police.

President Barack Obama has confidence in Secret Service director Mark Sullivan and believes he acted quickly to deal with a scandal in Colombia where agents were alleged to have engaged in misconduct involving prostitutes, White House spokesman Jay Carney said.

He added that the incident was being investigated and declined to speculate about conclusions that could be reached as a result of that probe.

The chairman of the US Joint Chiefs has said that the military is 'embarrassed' by the alleged military involvement in the misconduct scandal in Colombia, stating that "we let the boss down".

Speaking in a news conference, chairman Martin Dempsey said the substance of the Summit of the Americas had been overshadowed by allegations of misconduct by members of the Secret Service and military, which the Colombian police said involved prostitutes.

A prostitution scandal involving US security personnel in Colombia is threatening to eclipse President Barack Obama's charm offensive to Latin America.

In a major embarrassment for the United States at the Summit of the Americas attended by more than 30 heads of state, 11 US Secret Service agents were sent home and five military servicemen grounded over "misconduct" allegations in a hotel.

A US lawmaker who heads a congressional committee that oversees the Secret Service told CNN the incident apparently involved 11 agents "and they did bring women back to their rooms."