CARTAGENA, Colombia - A dozen Secret Service agents with President Barack Obama at an international summit have been relieved of duty because of allegations of misconduct.

The Associated Press received an anonymous tip that the misconduct involved prostitutes in Cartagena, Colombia, the site of the Summit of the Americas. A Secret Service spokesman would neither confirm nor deny the allegation.

A U.S. official, who was not authorized to speak publicly on the matter and requested anonymity, put the number of agents at 12. The agency was not releasing the number of personnel involved.

The incident threatened to overshadow Obama's economic and trade agenda at the summit and embarrass the U.S. The White House had no comment.

Secret Service spokesman Ed Donovan would not confirm that prostitution was involved, saying only that there had been "allegations of misconduct" made against Secret Service personnel in the Colombian port city hosting Obama and more than 30 world leaders.

Donovan said the allegations of misconduct were related to activity before the president's arrival Friday night.

Obama was attending a leaders' dinner Friday night at Cartagena's historic Spanish fortress. He was due to attend summit meetings with regional leaders Saturday and Sunday.

Those involved had been sent back to their permanent place of duty and were being replaced by other agency personnel, Donovan said. The matter was turned over to the agency's Office of Professional Responsibility, which handles the agency's internal affairs.

"These personnel changes will not affect the comprehensive security plan that has been prepared in advance of the president's trip," Donovan said.