US swimmer Ryan Lochte holding a press conference in Rio de Janeiro, two days ahead of the opening ceremony of the Rio 2016 Olympic Games. (AFP photo)

Four US Olympic swimmers have been accused by Brazilian authorities of lying about a robbery incident to hide a dispute over a damaged petrol station door in the city of Rio de Janeiro.

A Brazilian judge ordered Wednesday that multiple gold medalist Ryan Lochte and his three teammates Gunnar Bentz, Jack Conger and James Feigen have their passports confiscated for further police questioning.

The men claimed on Sunday that a taxi driver robbed them at gunpoint during a stop at a gas station.

However, the story turned out to be a lie when a surveillance video showed one of the athletes breaking the door to the bathroom and refusing to pay the damage.

The row ended when security guards were called in and the Americans reportedly paid the damage and left.

An image grab of surveillance footage showing US athletes leaving a gas station restroom in Rio.

Following the judge’s order, Bentz and Conger were taken off a US-bound plane at Rio’s airport on Wednesday night while Feigen was also forced to remain in the country. However, Lochte had already returned to the US earlier in the day.

“They were released by local authorities, with the understanding that they would continue their discussions about the incident,” said US Olympic Committee spokesman Patrick Sandusky.

He added that the swimmers were cooperating with local authorities involved in the case.

This is while, according to BBC, the swimmers have repeatedly changed their accounts of the incident and are likely to be charged with falsely reporting a crime.

Lochte repeated his robbery claims in an interview with NBC on Wednesday night, saying that when they used the restroom and went back to their taxi, the driver refused to move.

That’s when, he said, two men with guns and badges approached the cab and told them to get out and get down.

“I wouldn’t make up a story like this nor would the others – as a matter of fact we all feel it makes us look bad,” he claimed, while admitting that there were some inaccuracies in his original account.