(CNN Money) -- Congress was able to pass legislation to keep broader middle class income taxes from rising. But workers will still have to pay at least 2% more in payroll taxes.

That's because the government had temporarily lowered the payroll tax rate
in 2011 to 4.2% from 6.2%, in an effort to keep more cash in the
pockets of Americans and provide a boost to the economy. The tax cut,
which applies on the first $113,700 in annual earnings, expired on
Monday.

Now most of the country's 160 million workers
will see smaller paychecks. No one is expecting the payroll tax cut to
be extended.

Monthly paychecks will have $50 less for those
earning $30,000 annually, and will shave off $189.50 for those with
incomes totaling $113,700.

The amounts could be large enough to deal a blow
to middle class Americans' spending budgets, especially at a time when
the economic recovery is still struggling to gain a foothold.

Already Americans are feeling jittery. The latest reports show that consumer confidence took a dive
in December as fiscal cliff fears weighed on the country. A closely
watched consumer confidence index, which measures the American public's
sentiments every month, sank six points. It was major turnaround from
October, when the index reached a four-year high.

It was also the largest drop in consumer
confidence since the nation's credit rating was lowered in August of
last year.

Concern over these tax hikes are already holding shoppers back.

During the holiday shopping season, between Oct.
30 to Dec. 24, shoppers spent just 0.7% more than they did last year,
according to a MasterCard Advistors SpendingPulse report released last
week. That's compared to the 2% sales growth in the same period in 2011.

President Obama's top economic advisers have
also warned that if Congress didn't prevent tax hikes on middle class
families, consumer confidence could decline, retailers could be hurt,
and jobs may be cut.

"As we approach the holiday season, which
accounts for close to one-fifth of industry sales, retailers can't
afford the threat of tax increases on middle class families," the report
said.

Payroll taxes are used to fund Social Security, and the tax cut has cost about $120 billion each year. The Treasury Department has made up for the difference with money from general funds.