Gordon Brown today launched a scathing attack on the Tories after they confirmed that they were still committed to raising the inheritance tax threshold to £1m.

The prime minister told MPs that David Cameron's decision meant that the Conservatives were prioritising the needs of "the few", because just 3,000 wealthy families would benefit.

Yesterday Kenneth Clarke, the shadow business secretary, described the inheritance tax policy as simply an "aspiration".

This morning, Alan Duncan, the shadow leader of the Commons, told BBC Radio 4's Today programme that raising the inheritance tax threshold to £1m would be in the Conservative party manifesto, and that the party would introduce it in the first term of a Tory government.

A few hours later, in a statement to MPs on the outcome of last week's EU summit, Brown seized on the policy as evidence that the Tories were only interested in the wealthy.

Referring to Cameron's announcement last week that he would freeze the TV licence fee, Brown said: "Their policy is a 6p cut a week in the licence fee [for the many] and a £200,000 cut in inheritance tax for the few."

Brown went on: "Usually, at times of difficulty, the few who have money help the many. With the Conservative party, in times of difficulty the many should come to the aid of the few."

The prime minister also pointed out that Clarke was not in the Commons to listen to Cameron reply to Brown's statement. Clarke was "out of sympathy with the Conservative party", Brown claimed, and the Conservative party was "out of touch with the rest of the country".

In his Today programme interview Duncan said that the policy would be in his party's manifesto. But he also made it clear that the Tories were not committing themselves to raising the threshold in their first budget, suggesting that the policy had effectively been downgraded.

"We intend to make sure that only millionaires pay inheritance tax and, if we are elected, it will be in our manifesto and we will reduce inheritance tax during that parliament," Duncan said today.

The Tories would pay for the cut by introducing an annual charge on those who apply for "non-dom" tax status. At the time the policy was announced in October 2007 it was estimated to cost £3bn, but Duncan said it would be cheaper now because property prices were going down.

He also said it was consistent with cutting debt, which the Tory leader said last week would be the priority for his government.

"David Cameron said that one of his main priorities would be reducing this country's debt. That means public and personal," Duncan said.

"The thing about inheritance tax is that it is a tax on savings. What you will get with a better inheritance tax policy is people who will save and spend money in their old age and pass it on."

Following Clarke's comments yesterday, George Osborne, the shadow chancellor whose pledge on inheritance tax is believed to have persuaded Brown to abandon an early general election in the autumn of 2007, embarked on a damage limitation operation.

His spokesman insisted the pledge would be in the party's manifesto for the next general election, though Osborne could only commit to introducing it within the next parliament if the party wins.

Tory sources said the measure would be paid for by taxing "non-doms", or non-domiciled UK residents, though they admitted they would not know how much money this would raise until they formed a government and opened the Treasury's books.

The Tories are finding themselves increasingly under pressure over tax as they prepare for the budget on 22 April. This will confirm that Britain's public finances have deteriorated sharply since the pre-budget report in November, raising questions about tax commitments.

Last week Cameron indicated that the wealthy would have to pay their "fair share" during the downturn. This was seen as an indication that he would not reverse a new 45% top rate of tax for people earning more than £150,000 from 2011. The Tories had dismissed the 45p rate when it was floated by Alistair Darling in last November's pre-budget report.

Clarke backed Cameron's new stance, dismissing criticisms from the mayor of London, Boris Johnson, and Lord Tebbit. But he indicated the new Tory thinking when he said that raising the inheritance tax threshold to £1m would not be a priority. "That's an aspiration we have retained because inheritance tax hasn't moved in line with inflation," Clarke told BBC1's The Politics Show.

"But I don't think we are going round any longer saying that this is something we are going to do the moment we take power.

"We are going to have to consider when we get in whether we can afford to do that, in my opinion," he said. "I would hope we would do [it] ... The highest priority is tackling the debt, doing our best to avoid any more tax increases."

Eight hours later, Clarke insisted he was in agreement with Osborne. "We are fully committed to raising the threshold for inheritance tax in the first parliament of a Conservative government. This measure will appear in the manifesto and I support it. We also all agree that George Osborne cannot write his first budget until we have seen what we have inherited."

Lord Mandelson, the business secretary, said: "Ken Clarke's comments have thrown George Osborne's tax plans into confusion. The thing about Ken is that he just cannot help but say what he thinks. On economic and taxation policy, as on so many issues, his views are nearer to the government's than to the Cameron-Osborne Tory party."