Theresa May has waded into the row over Boris Johnson’s claim that Muslim women in burkas resemble letter boxes and bank robbers, urging the former foreign secretary to apologise after he defied an order to do so by Tory chiefs.

The prime minister said she agreed with the Conservative party chairman, Brandon Lewis, that Johnson should say sorry for his remarks, which she acknowledged had caused offence among the Muslim community.

She urged people to be “very careful” about the language they used to discuss issues such as women wearing the burka, but stopped short of saying Johnson should have the whip withdrawn for his remarks.

After a meeting with the Scottish first minister, Nicola Sturgeon, May told reporters: “I am very clear and the government is very clear about our position on the issue of the burka, which is that women should be able to choose how to dress. It’s up to a woman to decide how to dress, it’s not up to other people to tell a woman how to dress.

“It’s right that we have discussion about issues like this, but in doing that we all have to be very careful about the language and the terms that we use.

“Some of the terms that Boris used in describing people’s appearance obviously have offended people, and so I agree with Brandon Lewis.”

The former foreign secretary is understood to view the instruction as an attempt to shut down debate on a difficult issue that should be tackled head-on.

Lewis’s intervention makes him the most senior Tory to publicly criticise Johnson.

Johnson’s comments, in response to Denmark’s introduction of a ban on burkas in public places, prompted an angry reaction from Muslim organisations and MPs, who accused him of stoking Islamophobia for political gain.

The former foreign secretary, who is believed to be on holiday in Europe, could not be reached for comment.

Earlier, Burt told the BBC that Johnson had been defending Muslim women’s right to wear the religious dress. But he added: “I would never have made such a comment. I think there is a degree of offence in that, absolutely right.

“What he was trying to make a serious point about is that the UK government will not enforce any kind of clothing restriction on anyone. I wish he hadn’t accompanied it with a comment that I certainly wouldn’t make and I think many people would find offensive, yes.”

Fiyaz Mughal, the founder of Tell Mama, which campaigns against anti-Muslim violence, said Johnson’s comments “clearly” amounted to Islamophobia.

“These are the kind of comments we have seen that have been made by extremist far-right groups and people who have been maliciously attacking Muslims, so clearly it does fit that bracket,” he said.

Mughal criticised the “sheer flippancy” of Johnson’s comments and Lewis for not doing more to tackle Islamophobia in the party. He suggested Muslims needed reassurance from Downing Street.

“That reassurance should be coming quickly and effectively. It’s now 24 hours that have gone by, the message that members of the Muslim community get is their concerns are not taken into account,” he said.

Lord Sheikh, the founder of the Conservative Muslim Forum, set up to encourage British Muslims to get involved in political life, said Johnson’s comments were “totally out of order”.

“It is not in good fun. It’s a joke but in very, very bad taste. A joke like this will harm the community relations,” he told BBC Radio 4’s World At One.

“I don’t know whether this is his agenda to get the leadership of the Conservative party. Is he using Muslims as a springboard?”

The shadow equalities minister, Naz Shah, said: “An apology isn’t good enough. Boris Johnson’s comments weren’t accidental, they were a calculated attack in a national newspaper, made weeks after he reportedly met with Steve Bannon.

“Clearly the Tory party has an issue with Islamophobia, but over 24 hours later, the prime minister is still yet to say a word. Theresa May must condemn Boris Johnson’s comments unequivocally and order an inquiry into Islamophobia in her party.”

In his column for The Telegraph on Monday, Johnson said Muslim women wearing burkas looked like bank robbers and that schools and universities should be entitled to tell students to remove them.

He said it was “absolutely ridiculous” that wearers should “go around looking like letter boxes”, and he would expect his constituents to remove them in his MP’s surgery.

However, Johnson said he did not support a blanket ban on the face veil in the UK. “You risk turning people into martyrs, and you risk a general crackdown on any public symbols of religious affiliation, and you may simply make the problem worse,” he wrote.

Johnson appeared not to have acknowledged the scale of anger caused by his language. He is understood to believe that his article was a balanced piece simply pointing out that women should be free to wear the niqab if they choose and should not follow other countries in seeking a total ban.

Denmark introduced a burka ban last week, with fines of about 1,000 krone (US$155), following similar moves in France, Austria and Belgium.