David Cameron will face a battle to secure parliamentary backing for any EU budget deal that falls short of a real-terms cut after he suffered his first major Commons defeat on EU spending.

Senior Conservative MPs, who stopped short of joining 53 Eurosceptic rebels in the division lobbies, served notice that they will turn against the government if Cameron refuses to harden his position – that it must at least be frozen in real terms – at an EU summit later this month.

Downing Street moved to reach out to the rebels by dispatching William Hague to declare that the government would "take note" after 51 rebel Tories – plus two tellers – joined forces with Labour to defeat the government by 307 votes to 294, a majority of 13.

The rebel amendment demanded that the next seven-year EU budget, which will run from 2014-2020, should be "reduced in real terms".

The rebels, led by Mark Reckless, managed to defeat the government after Ed Miliband and Ed Balls imposed a three-line whip on Labour MPs to vote with the Tory Eurosceptics, many of whom are committed to leaving the EU.

The vote is not binding on the government. But No 10 sources made clear that the prime minister would lay down a "red line" at the EU summit, which opens on 22 November, to reject a planned 5% increase in the budget to ensure that it rises only in line with inflation.

"This is a red line," the Treasury minister Greg Clark told MPs shortly before the vote. "We will deploy our veto if our conditions are not met."

But senior Tory Eurosceptics, who declined to support the rebels because they did not want to vote in the same lobby as Balls, said they would have no qualms about rebelling if Cameron refused to change his position at the summit.

The prime minister's negotiating position would allow the EU budget to rise in line with inflation, which would lead to a 2% increase. The EU budget will have to be approved by MPs.

"When a budget deal is put to the Commons I will vote against it if there is any increase in EU spending," one former Tory cabinet minister said.

Labour seized on the prime minister's first Commons defeat on a government motion. Balls said: "This is a humiliating defeat for David Cameron, which shows how weak and out of touch he has become … David Cameron has failed to convince his own backbenchers, just as he is failing to convince other European leaders."

But there were signs of a backlash among Tony Blair supporters, who believe Cameron was right to accuse the Labour leadership of "rank opportunism". Douglas Alexander, the shadow foreign secretary, was Europe minister in 2005 when Blair brokered the last seven-year EU budget, which involved an above-inflation increase and cuts to the British rebate.

Margaret Hodge, the Blairite former minister who chairs the Commons public accounts committee, was heard to describe the Labour vote as "hateful" as she prepared for a meeting of her committee. "I hate this vote. I do not want to do it. It's hateful," Hodge said. "I just think it's outrageous. I'm almost wanting to abstain."

One former Labour cabinet minister said: "The danger is that we are stroking a dangerous underbelly of Euroscepticism." Another former cabinet minister said: "I suppose I can just about stomach having to vote for this if this is about scoring a tactical hit on the government. But if this marks a strategic shift in our position on Europe, then I would be very worried."

Labour said its position was consistent. Its MPs voted in favour of a real-terms cut in the budget in July.

The defeat marks a humiliating start for the new government chief whip, Sir George Young, who has presided over a government defeat shortly after taking office. But No 10 and the Treasury had briefed throughout the day that they would lose by 20 votes.

Loyal Tory MPs issued dire warnings of the dangers of rebelling over an issue that undermined the last Tory government. Sir Tony Baldry, the MP for Banbury, said: "We simply cannot carry on with this sort of self-indulgence that we are seeing on the order paper today.

"If this party hopes to be in government after the next general election, it has just got to get a grip and start supporting the prime minister."

But rebels hailed a great victory. Peter Bone, the Tory MP for Wellingborough, said: "Parliament spoke for the people. There was enormous pressure on colleagues to vote with the government.

"It was a very significant victory for the people. It was because MPs have to face their constituents."

The vote shows that the prime minister, who suffered a larger rebellion on a backbench motion on an EU referendum last year, is struggling to impose his authority on a sizeable chunk of his party.

The warning from some Eurosceptics that they are keeping their powder dry until the substantial Commons vote to approve the eventual EU budget deal shows that he will have a tough hand to play at the summit.

The prime minister will tell Angela Merkel at a meeting next week that he faces intense parliamentary pressure to freeze the EU budget. But No 10 expects the German chancellor to say that she faces a more important challenge – saving the euro.

Nigel Farage, the leader of the UK Independence party, said: "I am delighted that the house voted with the country rather than with the government whips. It is outrageous that the prime minister was prepared to go to Brussels in November and argue for what he would call a freeze and the rest of us would call an increase in the amount of money removed from British taxpayers to be spent by the distant EU bureaucrats."