"While the President's budget protects this vital program, legislation passed by House Republicans would make deep, harmful cuts that would make it harder for millions of Americans who are already struggling to put food on the table," the report said.

More recently though, food stamp spending has leveled off and is even expected to fall next year as the economy improves, according to a recent report by the Center on Budget and Policy Priorities, a left-leaning think tank.

Most food stamps go to children, the elderly, or people with disabilities, according to the White House, which highlighted a 2011 survey. Households with children accounted for 70% of food stamp benefits. And children, alone, accounted for 45% of food stamps, according to the White House.

Others say cutting food stamps could hurt the economic recovery.

"It won't be helpful if they decide to cut back on food stamps next year, which is one of our most effective anti-poverty programs," said Ken Rogoff, a Harvard University professor and economist.

The House bill would cut $40 billion in funding over a decade, forcing about 14 million people out of the program by 2023, according to the nonpartisan Congressional Budget Office. The proposal would also make it tougher for families to qualify for food stamps.

House Republicans say their reforms would eliminate those who shouldn't be getting food stamps by closing loopholes that let "people whose monetary resources are above eligibility standards into the program," according to a memo from the Office of House Majority Leader Eric Cantor.

"(The program) serves a noble purpose to help Americans who have hit bottom, which is why we must make certain it is working in the most effective and efficient way," said House agriculture chief Rep. Frank Lucas, in a statement.

A Senate version of the farm bill would save $4 billion from food stamps, with provisions aimed at ending fraud and abuse.