The Jewish organiser of the "Make Victoria Great Again Rally" which shut down parts of central Melbourne on Sunday says he's "disgusted" at being labelled a "Nazi" by hard-left counter-protesters.

Avi Yemini and 50 or 60 of his supporters were met with a much larger contingent of demonstrators – among them "Antifa" affiliates – when they fronted up in the CBD to criticise the state's handling of law and order issues.

Yemini, an Israeli Defence Force veteran and krav maga gym owner with political aspirations, gave a stump speech of sorts criticising Premier Daniel Andrews for being too soft on violent criminals.

"We all deserve the right to own our own businesses without groups of thugs coming in and robbing us," he said.

Despite leading the crowd in a chant of "black, white, straight or queer, we refuse to live in fear," the "From Charlottesville to Melbourne: Unite to fight the far right" rally accused Yemini of surreptitiously targeting Africans and Muslims with his tough-on-crime rhetoric.

That group believes Yemini is attempting to tap into the racist zeitgeist Donald Trump has been accused of exploiting in the US, and proceeded to label him a "Nazi".

Antifa-types also seized on the odd fact that some of Yemini's views are roughly aligned with those of some members of Australia's extreme right fringe, including Neil Erikson, who was seen at Sunday's rally taunting counter-protesters with a bullhorn.

When asked whether he was concerned by Erikson's presence – the former neo-Nazi pleaded guilty in 2014 to making threatening phonecalls to a Melbourne rabbi – Yemini said "he has nothing to do with me".

Regarding a recent Facebook live interview he conducted with Erikson outside the Melbourne court where the "patriot" was standing trial over the mock beheading of a dummy in Bendigo in 2015, Yemini said "I believe everyone has the right to free speech".

"If I disagree, I'll say so," he added.

Yemini, who commands a strong Facebook following and has appeared on Sky News's The Bolt Report, says he is keeping his political options open, and has not ruled out having a tilt on a One Nation ticket.