US president Barack Obama with Guatemala's president Otto Perez during a Summit of the Americas photo session, in Cartagena. Photograph: Eitan Abramovich/AFP/Getty Images

President Barack Obama said he would be "angry" if accusations that his security staff hired prostitutes at a key political summit in Colombia proved true.

Speaking for the first time about the scandal, Obama said agents represent the United States and are supposed to conduct themselves with the highest levels of dignity.

"Obviously, what's been reported doesn't match up to those standards," Obama said in a news conference wrapping up his appearance at the Summit of the Americas meeting in Cartagena.

The prostitution scandal has overshadowed Obama's trip. It came to light after US secret service agents allegedly took prostitutes back to their hotel in Cartagena before Obama arrived for the meeting this weekend. Eleven agents have been sent home and put on administrative leave while the scandal is investigated.

On Saturday the US defence department said a further five military personnel, staying at the same hotel, had been confined to their rooms after violating a curfew and "may have been involved in inappropriate conduct," .

A written statement said: "The conduct is alleged to have occurred in the same hotel where the recalled US secret service agents were staying."

The scandal has proved a major embarrassment for the White House at a meeting where Obama was discussing trade and the economy with 32 other heads of state. The president has come under fire at the meeting overthe impact of US monetary policy on the region and its attitude towards Cuba.

"I expect that investigation to be thorough, and I expect it to be rigorous," Obama said. "If it turns out that some of the allegations that have been made in the press are confirmed, then of course I'll be angry … We are representing the people of the United States, and when we travel to another country, I expect us to observe the highest standards."

Republican representative Darrell Issa called for a wide ranging investigation on Sunday. He told CBS's Face the Nation programme that there may have been more agents involved.

"The investigation will not be about the 11 to 20 or more involved, it will be about how did this happen and how often has this happened before," Issa told CBS. "Things like this don't happen once if they didn't happen before."

Peter King, chairman of the Homeland Security Committee, told NBC on Sunday: "This was wrong from the beginning to the end. If the facts are what they seem to be, it cannot be tolerated."

Prostitution is legal in Colombia in designated "tolerance zones". Local police were called to the Hotel Caribe in Cartagena's upmarket Bocagrande neighbourhood after a dispute with a woman in the room of one of the agents.

According to media reports, the agents had been drinking heavily before the president's arrival and taken women back to their hotel last Wednesday night. In the morning the woman involved in the dispute "freaked out" after she was not paid and banged on walls and doors in the hotel hallways.

King, who has been briefed on the situation, described a calmer scene to the Washington Post, which broke the story. He said that under hotel policy, overnight guests must leave photo identification at the front desk and leave the hotel by 7am the next morning.

According to King, the woman at the centre of the dispute had not left the hotel by 7am. Hotel officials called the police when the agent refused to open his room.

The woman refused to leave until she was paid, the agent disputed that he owed her any money. The incident was reported to the US embassy and then referred to the secret service.

The agents were recalled and replaced before Obama's arrival in the city on Friday afternoon. King told the Washington Post that "everything they did was a violation of proper conduct".

"First of all, to be getting involved with prostitutes in a foreign country can leave yourself vulnerable to blackmail and threats," King said. "To be bringing prostitutes or almost anyone into a security zone when you're supposed to protect the president is totally wrong."

The Hotel Caribe is less than 1,000 metres from the Cartagena Hilton where Obama was staying.

Secret service assistant director Paul Morrissey said in a statement that the situation had "no impact on the secret service's ability to execute a comprehensive security plan for the president's visit to Cartagena".

"This incident is not reflective of the behaviour of our personnel as they travel every day throughout the country and the world performing their duties in a dedicated, professional manner," he said. "We regret any distraction from the Summit of the Americas this situation has caused."

The incident is the biggest embarrassment for the US secret service since 2009 when two aspiring reality TV stars gatecrashed Obama's first state dinner at the White House and were photographed with the president.

Michelle Bachmann, former Republican presidential candidate, said:"Americans should be outraged." She told NBC's Meet The Press that the White House had "clearly been embarrassed by this".