David Cameron has today defended the Conservative alliance with a controversial Polish party in the European parliament, insisting it was not homophobic.

In an interview with Sky News, the Tory leader denied that the Law and Justice party – which opposes gay marriage – was prejudiced towards gay people.

Asked how he would respond to a letter from celebrities including Stephen Fry, which called on him to oppose "vile rhetoric" about gay rights from the Law and Justice party, Cameron said: "I'll respond to the letter very clearly.

"I don't believe the Law and Justice party are homophobic. Poland is a Catholic country. Most of the parties in Poland do take a stance like that on issues like gay marriage.

"I don't happen to agree with them, but they're not a homophobic party."

Last nigh, Ben Summerskill, the chief executive of the gay rights group Stonewall, pulled out of a gay pride event at the Tory conference in protest at appearances by Michał Kamiński, the head of the Law and Justice party and the leader of the Tories' new bloc in the European parliament, and Roberts Zile, the leader of the Latvian For Fatherland and Freedom party, another member of the new group.

"There is no doubt the progress that has been made in the last couple of years has genuinely been historic," Summerskill told Channel 4 News, referring to Tory attitudes to gay rights. "But the event tonight has been overshadowed by the presence, not just at conference but on the same platform as some senior members of the party, of people of such extreme and offensive views."

Cameron's alliance came under renewed pressure as the representative body of British Jews wrote to the Tory leader raising questions about their partnership.

A letter was sent from Vivian Wineman, the president of the Board of Deputies of British Jews, in a sign of the Jewish community's concerns over Cameron's pact with the Polish and Latvian parties.

In response, the Tories appeared dismissive about the discontent, saying in a statement that the board had been swayed by "politically motivated allegations made by the Labour party".

Kamiński and Zile both appeared and defended their political careers at a fringe meeting at the Tory conference in Manchester yesterday.

The pair received only gentle questioning, and Kamiński had been expected to speak at a late afternoon event held by the Conservative Friends of Israel, but did not appear. He did, however, earlier attend lunch with the group, where the Israeli ambassador to the UK, Ron Prosser, was present.

Last week, the board said it had looked into Kamiński and found nothing objectionable in his background, but its decision to write to Cameron suggests misgivings had become substantial.

The foreign secretary, David Miliband, said at the Labour party conference that Kamiński had an "antisemitic past" and highlighted the fact that members of Zile's party take part in annual celebrations of a Latvian unit of the Waffen-SS.

Zile said: "What has surprised me on this issue was I never thought this would come from a western democratic party. I would expect it to come from Moscow or the Kremlin, as it does from time to time.

"What happened was during the second world war [Latvian] people were often conscripted against their will to fight for both sides, Germans and Russians. Once a year these people commemorate their war dead, the people they fought alongside. They don't in any way commemorate Hitler or the Nazi regime. We would never do business with anyone who we thought glorified the SS."

The shadow minster for Europe, Mark Francois, who appeared on the same platform as Kamiński at a fringe meeting yesterday, said: "This is a slur which comes from the Soviet era, that was thrown by the Soviet authorities at the Latvian people. They in no way whatsoever commemorate anything to do with Hitler or the Nazi regime. The Latvian ambassador in London has chided the foreign secretary for using this tack, which basically dates from the days of Soviet propaganda."

Francois added: "All the parties in the group are signatories to the Prague protocol and we are perfectly happy with all of our partners that they believe in a liberal society and full respect for human rights."

Although the fringe event was carefully stage-managed – terse political lines trotted out and limited time for questions – there was one unfortunate mistake. The basement room in which delegates gathered to hear the controversial Tory allies was in Manchester's Midland Hotel, a building Hitler is said to have liked so much that he would have made it his northern residence if he had invaded.

Earlier in the day, comedians Eddie Izzard and Stephen Fry, both Labour supporters, published a letter condemning the Tories' alliance. The views of the Poles and Latvians were, they said, at odds with the Conservatives' new social liberalism.

A Tory spokesman said: "We're concerned that the politically motivated allegations made by the Labour party and their allies have been noted by the Board of Deputies.

"All these allegations have repeatedly been shown to be false. We will talk to the Board of Deputies about the lack of evidence behind them and their partisan nature. People should take account of the fact that the Polish chief rabbi has said that his remarks have been misrepresented, that the Latvian foreign minister has called David Miliband about his remarks, that the Latvian ambassador in London has spoken to the Foreign Office about the matter."