Some of those health workers might be the only doctor or pharmacist around for miles in a remote area and the prime minister was saying they were taking jobs so that Australians have to "clean the toilets" or the hospitals, he said.

"I think it really undermines the system," Prof McDonald said.

"The prime minister has talked about health workers for example in, I think, quite a nasty way."

Another federal government adviser on skilled migration, Michael Easson, told ABC television he would like to see the government or anyone else give up any evidence of any rorting.

The coalition says the 457 visa program is a core part of the migration system and Labor is playing to xenophobia in the community in order to attract votes.

But Ms Gillard told the ACTU conference she offered absolutely no apology for putting the opportunities of Australian working people first, front and centre, wherever they were born.

Temporary overseas worker numbers are up 20 per cent compared with the same time last year, while employment growth in the same period was around one per cent.

"The number of people coming here to fill short-term gaps should not be growing twenty times faster than employment overall," Ms Gillard said.

The government's policy changes will include forcing employers to demonstrate they are not nominating positions where a genuine shortage does not exist, raising English language requirements and raising the market salary exemption from $180,000 to $250,000.