The peer – Mr Cameron’s tennis partner and an Oxford pal – denied making the comments and today will be grilled by the party’s board to get to the bottom of the row.

It came as rebels on the right prepared to side with Labour and Lib Dems in a crunch vote on extending civil partnerships to straight couples.

It would be a catastrophic Commons defeat for Mr Cameron.

The change would cost billions and delay by 18 months the gay weddings he has championed.

Lord Andrew Feldman (Image: Rex)

Loongate was sparked after an anonymous member of the PM’s inner circle dismissed last week’s rebellion by 116 MPs over an EU referendum, blaming it on pressure from ­constituency parties.

At a Westminster restaurant with journalists, the figure said: “The MPs just have to do it because the associations tell them to and the associations are all mad, swivel-eyed loons.”

Lord Feldman was named online as the man behind the comments, and he has now threatened to sue.

He issued a statement insisting he did not make the “derogatory comments”. He said: “I did not, nor have I ever, described our associations in this or in any similar way.”

David Cameron (Image: Getty)

The PM has given his old university friend his personal backing and senior party figures including Health ­Secretary Jeremy Hunt were sent out to praise him yesterday.

But one member of the board that will quiz Lord Feldman at this afternoon’s meeting openly ­questioned his denial.

MP Brian Binley said: “I don’t want to pre-judge the matter but it appears it was said.

“I do not believe journalists of this kind lie. I assume something was said. I don’t know whether it was said as a joke but if it was it belies an attitude of mind.”

The right-wing backbencher added: “Something of this sort must have been said. We need to know the truth.”

David Mellor (Image: Getty)

Tory ex-Cabinet minister David Mellor weighed in too, saying if Lord Feldman did not make the remarks he should follow through on his threat to sue.

Mr Mellor told Sky: “He strenuously denies it but, if so, I have to say as a former lawyer… sue them. I think we will find the writ will not appear.

“If it was him this has been a disaster waiting to happen, because you cannot elevate tennis-playing friends to be chairman of the party without there being a political price to pay.”

Perhaps most tellingly, former party chairman Norman Tebbit said the main problem was not whether the comment had been made, or who it had been made by, but that many people in the party did not doubt it was said. He added: “People think it has the ring of truth.”

Lord Tebbit accused Mr Cameron of abandoning the “common ground” of British politics by championing issues such as gay marriage and said he should give peers the green light to dump it.

Lord Norman Tebbit (Image: Getty)

The grandee said: “He could whisper in the ears of the whips in the Lords that he would not burst into tears if he thought it would go down the swanney.”

The attack came after former ­Chancellor Lord Geoffrey Howe said on Saturday that the PM had also lost control of his party over Europe and was “running scared” of this backbenchers.

Bob Woollard, chairman of the Conservative Grassroots organisation, headed a group of more than 30 serving and former constituency association chairmen who handed a letter in to Downing Street yesterday calling for gay marriage plans to be scrapped.

He demanded: “Do I look like a mad, swivel-eyed loon? I don’t know who said that, but whoever said that, if it was somebody close to David Cameron, the Prime Minister needs to disassociate himself from those words, reprimand the person and if the person isn’t prepared to give a fulsome apology, sack them.”

Former Chancellor Lord Geoffrey Howe (Image: Getty)

He said same sex marriage was a “tipping point” that has forced many to say: “I’ve had enough, I’m off, I will never vote Conservative again.”

UKIP has also seized on the “swivel-eyed” jibe. Its leader Nigel Farage – who himself quit the Conservatives in 1992 over Maastricht – has called on other disgruntled members to join him.

Up to 200 Tories are set to defy Mr Cameron on gay marriage today, including at least two Cabinet ministers, Owen Paterson and David Jones, and probably a third, Defence Secretary Philip Hammond. Justice Secretary Chris Grayling is expected to back amendments that will let public sector workers refuse to carry out gay marriages.

Traditional right-wing Tories see the vote as their best chance of killing off same-sex weddings, which are due to start next summer. Any changes would mean Whitehall and Town Hall computer systems having to be updated.

The Treasury would also face finding an extra £4billion in public sector pension payments, a cost that will be even greater for private firms.

Tory MP Jackie Doyle-Price yesterday said that while she had always “been in favour of marriage equality” she could not support the gay marriage Bill because it was such “a mess”.

The Prime Minister has threatened to collapse the Coalition as he ­struggles to deal with his Lib Dem ­partners as well as his own MPs. Mr Cameron told Total Politics: “What matters to me, is can we get things done? Can we improve the state of the country? Can we fulfil our manifesto? “The best way to do that is to continue with the Coalition, but if that wasn’t the case we’d have to face the new circumstances in whatever way we should.”