The National Taxpayer Advocate found only 61 percent of people seeking to speak with a customer service representative last year got through to anybody -- leaving nearly 20 million calls unanswered.

The report largely blamed budget cuts, and lamented the impact the poor customer service is having on taxpayers.

"At the risk of vast understatement, it is a sad state of affairs when the government writes tax laws as complex as ours -- and then is unable to answer any questions beyond 'basic' ones from baffled citizens who are doing their best to comply," the report from National Taxpayer Advocate Nina Olson said.

The study detailed how customer service has steadily declined over the past several years, including at its 400 "walk-in sites." In fiscal 2014, the office said, the IRS will only answer "basic" questions at those sites during filing season. And it will not answer any questions, "even basic ones," after April, even for filers who got extensions.

"In addition, the IRS will discontinue its longstanding practice of preparing tax returns for low income, elderly and disabled taxpayers who seek help," the report said.

With the erosion in services, wait times have gone up. In fiscal 2004, callers were left on hold for just 2.6 minutes. Today, the average wait time is nearly 18 minutes.

Some taxpayers resort to writing letters to the IRS with their questions. The agency received 8.4 million such letters last year, but more than half were not answered by the end of fiscal 2013, the report said.

The problems stems in large part from budget cuts. The training budget has, according to the report, gone from $172 million a few years ago to $22 million. The workforce itself has fallen from 95,000 full-time workers to 87,000.

"Thus, the IRS not only has fewer employees than four years ago, but those who remain are less equipped to perform their jobs," the report said.

As the IRS is reluctant, like other agencies, to simply lay off employees in the face of budget cuts, it has targeted areas like training for cuts.