A close ally of Nick Griffin, leader of the British National party, has been forced to withdraw from speaking at a London conference in support of Palestine amid mounting fury from anti-fascist groups.

Several politicians due to speak at the conference, organised by the Palestinian Return Centre (PRC), were alarmed to learn the background of fellow speaker Krisztina Morvai, a Hungarian MEP and leader of the far-right Jobbik party in the European parliament.

The fledgling party, which recently formed an alliance with the BNP, has attracted fierce criticism for its links to a grassroots militia and the controversial views expressed by some of its supporters towards gay people and and Jews.

But it is Jobbik's attitude towards Hungarian Gypsies that has attracted the most controversy. During this year's European elections, the party blamed the country's Romany population for the decline in Hungary's living standards, and says on its website that "voters have had plenty enough of Gypsy crime".

The Hungarian Guard militia, which is backed by Jobbik, has marched through Romany ghettoes in Hungary, increasing community tensions. Militia members wear uniforms emblazoned with a striped red-and-white symbol, a version of which was used by the Arrow Cross, a pro-Nazi party that ruled Hungary for a brief but brutal period towards the end of the Second World War. Morvai has been photographed wearing the uniform of the militia, an organisation ruled illegal by a Hungarian court and described as "Hungary's shame" by a former prime minister of the country, Ferenc Gyurcsány.

Labour MP Neil Gerrard pulled out of next month's PRC conference after learning of Morvai's background. The Liberal Democrat peer Baroness Tonge, independent MP and former Labour minister Clare Short, and Daud Abdullah, deputy secretary of the Muslim Council of Britain, who are all due to speak at the event, were also alerted about Morvai's role in Jobbik.

The conference is promoted by the European Campaign to End the Siege of Gaza. A spokesman for the campaign confirmed that Morvai's invitation had been withdrawn following protests. "We did not want to get into trouble, so we decided to exclude her," he said. "We want the conference to be about raising awareness about Palestine."

The move has been welcomed by critics of the BNP, who have expressed alarm about the blossoming relationship between it and Jobbik. Earlier this month the Observer reported how the BNP and Jobbik were forming a new alliance in the European parliament that will include France's Front National and a handful of other far-right groups. Griffin spoke at a Jobbik rally last year and has met several of its senior officials.

"I am pleased to hear the PRC have withdrawn their invitation to Jobbik's Krisztina Morvai," said James Bethell, the director of Nothing British, a group that campaigns against the BNP. "She is one of Europe's leading neo-fascists, a senior member of Nick Griffin's political alliance and an MEP for a revolting party."