The Islamic Council of Victoria will boycott a planned meeting with Prime Minister Tony Abbott in Melbourne on Tuesday, saying his comments about migrants being part of Team Australia "indicated the meeting was a media stunt".

Council secretary Ghaith Krayem said on Tuesday that the organisation was upset at comments by Mr Abbott on Sydney radio on Monday that "you don't migrate to this country unless you want to join our team".

Mr Krayem told radio station 774 ABC Melbourne that the council had initially agreed to attend the meeting but pulled out after hearing Mr Abbott's comments.

The leaders used the bridge-building meeting to raise doubts about creating no-go zones that would force travellers to Iraq and Syria to prove they are not terrorists, but broadly welcomed the government's initiative and praised Mr Abbott's efforts

Mr Krayem said the language used by Mr Abbott was not befitting a leader of a country and echoed those of right-wing extremists.

He said said the council wrote to Mr Abbott after the government dumped the planned repeal of race-hate laws and announced enhanced national security laws detailing its concerns, but said the organisation received no reply until the invitation to attend Tuesday's meeting.

"The government has already decided what they want to do and they are looking for a rubber stamp," he said.

The council released a statement saying: "What we look for from our Prime Minister is true leadership in difficult times but what we find ... is a Prime Minister who wilfully places our community in the crosshairs of every bigot and racist in the country.

"When we are trying to build harmony and goodwill across the community we cannot understand the motives behind the Prime Minister’s inflammatory remarks.

"Accordingly, the ICV has withdrawn its attendance from today’s meeting and urges all other community leaders to do likewise."

During the radio interview, Mr Krayem rejected suggestions from the Prime Minister that moderate members of the Muslim community should speak out against radical, saying his organisation had been vocal about the issue.

"We as community leaders want to ensure the security and prosperity of our nation. We are Australians," he said, adding the relationship had to be genuine and Mr Abbott's language did "not reflect that".

"The language of leaders has an absolute influence on what the person in the street says and does," Mr Krayem said, pointing also to Attorney-General George Brandis' comments about the "right to be a bigot".

He said the Prime Minister's comments could add to what Mr Krayem said was a rise in Islamophobia in Australia.