Peter Dutton has hit back at Federal Greens MP Adam Bandt who labelled the Immigration Minister a "terrorist" at a rally in Melbourne protesting against the treatment of men who remain on Manus Island.

More than 1,000 people gathered at the State Library before staging a mass sit-in at Federation Square to show their support for 600 refugees and asylum seekers who remain at the Manus Island detention centre in Papua New Guinea, despite its official closure on Tuesday.

Trams and traffic were brought to a standstill, while access to the train station was disrupted. Hundreds also gathered at a similar rally in Sydney.

'The Government's position is absolutely clear'

The UN High Commissioner for Human Rights said there were serious concerns about the safety and wellbeing of the men, and both Australia and PNG were responsible under international human rights law to protect them.