Since 2010, the agency's budget has been reduced by almost $1 billion and around 10,000 employees have been cut. Under the 2014 budget, the IRS will receive $11.3 billion -- nearly $2 billion less than the White House had requested for the agency and a $526 million drop from 2013.

Meanwhile, government spending cuts last year forced the IRS to furlough workers without pay for threedays, making it even harder for the agency to keep up with its workload.

To cut costs, the IRS has been conducting more audits by mail than in person. Last year, more than three-quarters of examinations were correspondence audits, and the rest were field audits -- meaning they were conducted at an IRS office or a taxpayer's home.

And while the overall number of audits was low, at around 1% of all taxpayers, there are still certain groups that aren't getting a break.

One of those groups is the rich: About 9% of taxpayers with income between $1 million and $5 million were audited last year, and that rate rose to 16% for those with income between $5 million and $10 million. For the nation's top earners, with income over $10 million, the audit rate was 24%.

Business owners are also more likely to be audited, and so are taxpayers who claim a home office deduction or the Earned Income Tax Credit. Reporting -- or failing to report -- a foreign bank account could also lead to additional scrutiny, as the IRS continues to crackdown on people hiding offshore income.

In addition to being unable to conduct as many audits, the agency's taxpayer assistance has been deteriorating, the National Treasury Employees Union said in a statement Friday.

"We are seeing the results of these reductions in staffing, particularly in customer service, all across the country," NTEU president Colleen Kelley said. "Both taxpayers and employees are frustrated by the lengthy lines at Taxpayer Assistance Centers and the long telephone hold times for those who call the IRS with a question."