The Pride flag was actually put up on a "courtesy" flag pole which otherwise flies the City of Toronto flag. Multiple flag poles around city hall fly the Canadian flag and the city flags at all times.

A city spokeswoman said the mayor didn't officially order the rainbow flag to come down, but did ask how it came to be raised.

"The mayor did ask to seek clarity on how this rainbow flag could be flown at city hall," said Jackie DeSouza, explaining that a non-profit group had asked to have the rainbow flag flown for the duration of the Sochi Games.

DeSouza said the group's request met the requirements of the city's flag flying policy, which was highlighted for the mayor.

"Given that city council approved this policy it would have to be up to council as a whole to change the policy," she said when asked if the mayor could order the flag to come down.

The flag flap drew some LGBT activists to the mayor's office to protest his comments.

"That he actually would go out of his way to remove a flag which is a symbolic gesture in support of those very basic human rights is probably more upsetting to me than anything he's done around here,'' Suzy Richter said after a conversation with the mayor.

Councillor Sarah Doucette told CP24's Katie Simpson she doesn't agree with Ford's position as there are already Canadian flags flying at city hall.

"If he's not prepared to change then I don't think he should be mayor," she said.

But Councillor Denzil Minnan-Wong said an Olympic or Canadian flag should be raised instead.