Calls to annihilate the IRS are gaining a lot more traction this week after the scandal involving agents singling out conservative and religious groups.

"The tax man shouldn't be a part of every decision that you make in your life," says columnist Theo Caldwell in an interview with KIRO Radio's Dori Monson Show. "And now I think is the time on a bipartisan basis to have that discussion."

Caldwell, in a widely shared post in The Daily Caller, argues the IRS has far overshot its mission to become the most powerful and feared institution in America.

"Should a nominally free country countenance the harassment of its citizens, at home and abroad, by Mr. or Mrs. Cough-It-Up who, having overcome the unspeakable tragedy of having no given name, nevertheless demands to know everything about you?" he writes.

Caldwell says there's something inherently wrong with a tax system indecipherable to all but the wealthiest people, who make millions by manipulating it.

And he argues the rest of us spend way too much time paying more than our fair share while clinging to measly deductions for charitable giving, our mortgages, and anywhere else we can save a few bucks.

"I think it is untoward, it's unbecoming as a free nation that the tax authority is the arbiter of everything that we do and say."

Instead, he advocates getting rid of the whole tax code and replacing it with a flat tax.

Under his proposal, anyone making over $50,000 would pay 15 percent of their earned income. And if that's too radical for some people to swallow, he suggests two or three rates on a progressive scale instead - something along the lines of 10-20 percent.

"No deductions, no investment or estate taxes, we're done."

That means "no deductions, no investment or estate taxes, we're done," he says.

That also means the break on your home or donation goes away.

"You don't have to worry about making donations or giving this and that based on whether or not the tax man is going to bless that. You want to give money to your church or synagogue, write a check," he tells Dori.

Proposals like Caldwell's have been met in the past with outrage and animosity (just ask Ron Paul.) But he says the latest IRS scandal opens the door for both Republicans and Democrats to come together and finally change the system.

While Caldwell insists he's a long time critic of President Obama and Democrats, his proposal is bipartisan.

"Democrats, Republicans , Libertarians, librarians, I don't care. You've got to be able to get behind the idea of just getting rid of the IRS," says Caldwell. "I'd love to hear from some disinterested party who can tell me that they think it's a good system now and we can't do better. I know we can."