Boris Johnson was plunged into his first big race row as London mayor last night when he was forced to sack his chief political adviser for having suggested that older African-Caribbean people should move to the Caribbean if they are unhappy living in a Tory-controlled London.

James McGrath was recorded on tape saying: "Well, let them go if they don't like it here."

Johnson announced late last night that McGrath, a 34-year-old Australian-born Tory high flyer, would leave City Hall with immediate effect after a tape recording emerged of his remarks to Marc Wadsworth, a journalist.

"It is with great regret this evening that I have accepted the resignation of my political adviser, James McGrath," the mayor said. "Unfortunately, his remarks in a conversation with an internet journalist, published this weekend, made it impossible for him to continue in that role."

The mayor continued: "London is blessed with a rich ethnic and cultural diversity. It is one of the main reasons why I regard it as the greatest city on earth, and I am determined, as mayor, to serve each and every community with equal passion and commitment."

Johnson consulted David Cameron, the Tory leader, on the issue. Johnson said last night that McGrath was not a racist, and he believed that he had run into trouble after a "silly and hostile suggestion" was put to him by Wadsworth.

This was a reference to a meeting last month at which Wadsworth asked McGrath his reaction to an article by the writer Darcus Howe on the eve of the mayoral election.

Howe had written that the election of Johnson might trigger a mass exodus of older African-Caribbean migrants back to the West Indies.

Wadsworth wrote on his blog that McGrath responded: "Well, let them go if they don't like it here."

Johnson said: "James's remark was taken out of context and distorted, but he recognises the need for crystal clarity on a vital issue like this.

"We both agree that he could not stay on as my political adviser without providing ammunition for those who wish to deliberately misrepresent our clear and unambiguous opposition to any racist tendencies."

The sacking last night represented a rapid U-turn by Johnson's camp - which at first had maintained that McGrath had done nothing wrong when his remarks were set in the context of hostile questioning from Wadsworth.

The Guardian was warned earlier in the day that it might face legal action from McGrath if it reported McGrath's remarks in an inappropriate way.

This initial firm response was directed by McGrath who issued a furious response to Wadsworth when he reported his remarks on the citizen journalist website (the-latest.com) on Friday.

McGrath resisted standing down until late yesterday, did issue a statement to Wadsworth confirming that he had made the comments but he insisted they were made in response to "inflammatory comments" by Howe, a supporter of Ken Livingstone. Describing himself as "an immigrant myself," the Australian-born McGrath wrote: "I felt that this suggestion [by Howe] was ridiculous and intended as a slur and responded by saying with words to the effect of 'let people go if they don't like it here'."

Wadsworth last night welcomed the resignation of McGrath. "James McGrath has done the right thing, though I had not called for him to resign. I just wanted a dialogue with him."