Federal Communications Minister Malcolm Turnbull joined Labor's Anthony Albanese and Greens Senator Sarah Hanson-Young at the media launch of the Bingham Cup in Sydney on Friday.

"It is important always to oppose homophobia... It has no place in Australian society and no place in Australian sport," said Mr Turnbull who was speaking in Rose Bay, in the heart of his Eastern Suburbs electorate.

During the media conference, Mr Turnbull's Liberal Party colleagues, Social Services Minister Kevin Andrews and Victorian Attorney-General Robert Clark, announced they would not be attending Saturday's World Congress of Families conference as it was being held by a far-right religious group in Victoria.

"As I have been informed today that the event is being hosted by Catch the Fire, I have decided not to attend," said Mr Andrews who was to receive a prestigious WCF honour, the 'Natural Famliy Man of the Year'.

The pro-life and anti-gay WCF had several venue cancellations until the Catch the Fire Ministeries stepped forward to host the Saturday conference, which is expected to be attended by protestors.

The high-profile withdrawals for the anti-gay conference came somewhat coincidentally on "Wear It Purple Day" — a national initiative to promote sexuality and gender diversity.

"I think government ministers, particularly government ministers who have responsibilities for families have a responsibility to recognize their ideal of a family mightn’t be the same as everyone else’s," said Labor's Mr Albanese to BuzzFeed.

Senator Hanson-Young was particularly scathing that it took so long for Mr Andrews — whose portfolio includes family welfare — to cancel.

"(He) probably should never have been a key host in the first place and booked key rooms in Parliament House for them. People expect better from our politicians," she told BuzzFeed.

The Bingham Cup organisers told BuzzFeed the support from all political figures has been remarkable, with US Ambassador to Australia, John Berry expected to attend the tournament on Sunday.