For many people, filing and paying their taxes can be a major source of anxiety every spring. On top of all the other things you're already juggling in your daily life, you've got those pesky tax stubs, the federal and state returns, and standing in line at the post office. And while many taxpayers are expecting a nice refund, there are plenty of folks out there who owe money to the IRS this year.

So what if you just don't do it? What really happens to the people who ignore the whole thing and say, the heck with all of this, I'm skipping tax season?

Well, here's the surprising news: not much. At least, not much happens to you at the beginning, and your chances of getting nabbed by the IRS — and having to pay up — differ significantly depending on the kind of income you've earned over the year. (See also: 15 Surprising Facts About Income Tax)

Which Is Worse — Filing Late or Paying Late?

Let's back up a second. The law requires taxpayers to both file a tax return and pay any taxes due. And there are separate penalties associated with each of your legal obligations every year.

But, contrary to what a lot of people think, the "failure to file" penalty is actually worse than the "failure to pay" penalty. And yes, the Code is way harsh and adds interest to the penalties as well as to the principal. Also, keep in mind — you're slapped with both the failure to file and the failure to pay penalties if you blow off doing both of them.

That's why, if you are facing a bill that you can't afford, tax professionals always tell their clients to at least file on time so they can avoid all of the late filing penalties and interest. And lawyers will advise you to pay any part of your liability if you possibly can, since the IRS accepts partial payments — and something is always better than nothing when it comes to your taxes.

Who Is Most Likely to Be Caught?

So who does the IRS catch up with the most? It's simple. People who've had a third-party report earnings to the IRS for that tax year.

For example, most of our employers are required to provide information to the IRS before they pay us. Brokers generally have to report gain on stock or bond sales. Even a casino will ask you to fill out paperwork before you take home a large jackpot.

And that's just the beginning. If you received unemployment, pension income, or pulled money from your IRA in 2011, you've left a paper trail that will lead the IRS straight to you.

The people who have the best chance of flying under the radar for the longest amount of time? Freelancers and those who are self-employed or earn most of their income from consulting or odd jobs, because the IRS has the hardest time tracking unreported money. At the end of the day, someone who works for herself is usually the only one who reports it, and therefore, the odds of getting busted are much lower than a full-time employee at a more traditional company.

But while it may take years for the IRS to hunt you down, and you may get lulled into a sense of complacency thinking the IRS has forgotten about this tax year, you're better off paying the piper sooner rather than later.

What Is the Best Way to Deal With IRS Problems?

In all the cases I've seen, the government has been more willing to work out a compromise or a payment plan with an individual when the person came forward on their own.

Sure, some Americans will get away with not paying the government a dime this year. But if and when they do come down on you, the IRS can be like a three-ton gorilla. I've watched people's wages garnished, their assets seized, and their lives turned upside down until their tax debt is paid in full.

So if you find yourself thinking, "Hey, maybe I'll forget about my taxes this year," make sure you're ready to gamble on one of the most powerful agencies in the country. And you can sleep at night knowing all those penalties and interest are out there, accruing on your account.

Like this article? Pin it!

Jacoba Urist is a tax and estate attorney from New York. She is a regular contributor to The Huffington Post and is writing a book "The Happiest Parent" about preparing the best possible future for every child. Follow her on Twitter at @TheHappiestPare.

Disclaimer: The links and mentions on this site may be affiliate links. But they do not affect the actual opinions and recommendations of the authors.

Wise Bread is a participant in the Amazon Services LLC Associates Program, an affiliate advertising program designed to provide a means for sites to earn advertising fees by advertising and linking to amazon.com.

What year was the tax law enacted that made it mandatory for citizens to pay taxes? Can you reference the actual tax law please? I heard that it was never a law to pay tax, and it sounds like something our forefathers would be rolling over in their graves about, much like our union itself. They never wanted a large central government.