Ordained as a Baptist minister in 1987, Costello, along with his wife Merridie and a team of others, rebuilt the congregation at the St Kilda Baptist Church and opened a drop-in centre.[1] As part of the church's outreach program, he started a legal office at the church where he practised as a part-time solicitor.[11] He also taught urbanministry at Whitley College, a college of the University of Melbourne.[11]

Costello was elected Mayor of St Kilda Council in 1993, and became well known for championing the cause of local democracy and for his clashes with Victorian PremierJeff Kennett.[1] Kennett attacked Costello as being un-Victorian for speaking out against the gaming-led recovery of the state and often referred to him as "that leftistcleric".[1] Costello's political career ended when his mayoral position was abolished with the whole St Kilda Council in Kennett's reform and consolidation of local government in 1994. Towards the end of that time, he was approached by the Australian Democrats to fill a casual vacancy in the Senate, but decided against it, partly due to the likelihood that this could place him in direct conflict with his brother if Peter became Treasurer.[16]

Throughout the late 1990s and early 2000s, Costello was frequently seen in the Australian national media commenting on gambling and other social problems.

Costello has spoken out in favour of stronger gun control in Australia, acting at times as the co-chairman or spokesman of the National Coalition for Gun Control.[17]