Half of those nonpayers earn too little to pay any taxes, and half of them get there through tax deductions and exemptions, according to the Tax Policy Center. Of the latter half, 44 percent use tax deductions designed to help the elderly, and 30 percent use tax deductions that aid the working poor or children.

The Tax Policy Center uses a family of four earning less than $26,400 as an example. After the $11,600 standard deduction and four $3,700 exemptions, they have no taxable income.

But many of these people pay federal payroll and excise taxes, as well as state income taxes.

The ranks of those who don’t pay income tax has swelled because of unemployment and underemployment during the recession and the sputtering recovery. More than 13 million Americans are jobless.

Also driving an increase: tax policies pushed by Republicans. The Earned Income Tax Credit, greatly expanded by Ronald Reagan in the mid-1980s, is designed to give money back to low-income workers to increase their incentive to work. And a 2004 Tax Foundation study found the Bush-era tax cuts erased the income tax for 7.8 million families by lowering rates and doubling the child tax credit.

Not all of those avoiding income tax are poor, however. Even 4,000 people earning more than $1 million managed to not pay income taxes in 2011 because of deductions, according to the Tax Policy Center.

After video of Romney’s comments surfaced Monday, the former Massachusetts governor stood by his remarks but admitted they were not “elegantly stated.”

Romney’s comments also grouped together the 46 percent of Americans who don’t pay income tax with people who receive government benefits. According to The Wall Street Journal, 49 percent of Americans lived in a household where at least one member of the family received a government benefit in 2011. There’s not a perfect overlap between these two groups.