Mr Kennett also ruled out a return to politics, telling Fairfax Radio that talk of such a move was "absolute stupidity".

Mr Baillieu is heard referring to independent MP Geoff Shaw, upper house Liberal MP Bernie Finn and his "crazy mates" and federal government minister Kevin Andrews in a taped recording between himself and a journalist from The Age.

Mr Kennett said the tape shows Mr Baillieu is unhappy in the job.

"The point is, why do you continue in the job when you're clearly not happy with many of those who work there?" he said.

Mr Kennett said any decision was up to Mr Baillieu, but added: "If you're no longer happy there ... move on."