Treasurer Wayne Swan says he did not immediately condemn "offensive" comments made at a union dinner as soon as they occurred because he did not want to give them another airing.

Mr Swan spoke at the function on Wednesday evening after a comedian hired by the Construction Forestry Mining and Energy Union (CFMEU) made the remarks about Tony Abbott and his female chief of staff, Peta Credlin.

He says the comments were offensive but did not condemn it until the next morning.

Mr Swan has told Lateline that was a mistake.

"I didn't want to give it oxygen," he said.

"I think in retrospect it would have been better if I did, but I didn't want to give it a further airing and I was the guest speaker, I was actually concentrating on my remarks."

Senior Labor figure Craig Emerson says he walked out in disgust when the comedian made the comments, but other Labor MPs stayed behind.

Mr Swan released a statement on Thursday afternoon, condemning the comments as "very inappropriate and offensive" and said he made his views clear to the union the following morning.

He later released a second statement, saying: "This was a comment made by a stand-up comedian which clearly crossed the line, which is why I conveyed my strong objections to the union this morning, though on reflection I should have done that last night."

In the wake of controversial comments made by Sydney broadcaster Alan Jones at a Liberal Party function several weeks ago, several senior Labor MPs questioned why no-one walked out of the dinner.

Dr Emerson says he left the dinner because he was "disgusted at the comments", but defended colleagues who chose to stay.

"It's a matter for each individual who was at that meeting, who was at the dinner, and the Deputy Prime Minister has condemned those comments," Dr Emerson said.

Cabinet minister Brendan O'Connor was also at the dinner and has described the joke as "offensive".

"I conveyed my feelings to the organisers of the function and I understand the union and the comedian in question have unreservedly apologised," he said in a statement.

The statement did not respond to questions of why Mr O'Connor did not follow Dr Emerson's lead and walk out.

Prime Minister Julia Gillard says she left the function before the comments were made, but phoned the union the next morning to express her condemnation.

"Let me be very clear; on the reports to me, the remarks that were made were deeply offensive, they're wrong, the comments should never have been made," Ms Gillard told reporters in Canberra.

Let me be very clear; on the reports to me, the remarks that were made were deeply offensive, they're wrong, the comments should never have been made.

PM Julia Gillard

"It was wrong for them to have been made and offensive for them to have been made.

"In those circumstances, as soon as I possibly could, I rang the national secretary of the CFMEU and indicated I thought these comments were offensive."

'Poor judgement'

The CFMEU says the comedian's material was not vetted and the union's leadership is "upset" by the comments.

The company behind the comedian has issued a statement apologising to Ms Credlin and Mr Abbott.

"The joke was a last-minute inclusion and crossed the line," Manic Studios said.

"We did not run the joke past anyone. Nobody from the CFMEU or the Labor Party or anyone else in attendance at the dinner knew of the joke until it was told.

"The poor reaction the joke received is testament to fact that we exercised poor judgement."

Workplace Relations Minister Bill Shorten was also at the dinner, but says he left before the comment was made.

"The remarks are tasteless. I don't want to give these tasteless, out of order remarks, offensive remarks, personal remarks, any more oxygen that they deserve," he said.

He said he had contacted the union the morning after to express his disappointment.