"You work your whole life to build up a nest egg or a family-owned business or family farm. Then you pass away. ... Uncle Sam can swoop in and take over 40% of everything you've earned over a certain amount. It's just wrong," Brady said last week at an event promoting tax reform.

But that assertion overlooks a few key details about the estate tax, which the Congressional Budget Office projects will raise between $25 billion and $34 billion a year over the next decade.

1. It only affects 0.2% of all estates today

Only those estates worth more than $5.49 million this year (or $10.98 million for married couples) even have to file estate tax returns. Thenonly about half of those end up being taxable after factoring in deductions and credits.

This year, the Tax Policy Center estimates that 11,310 estates will have to file a return, but just 5,460 will owe any federal tax.

Plus,the 40% tax rate that Brady mentioned only applies to the portion of an estate above the $5.49 million exemption level ($10.98 million for couples).

The federal estate tax used to be harder to avoid but even then it never touched most people.

Back in 2000, Americans could only exempt $675,000 at death, and those with taxable estates were subject to a 55% rate. That year, 52,000 estates ended up being taxable, according to the IRS, or about 2.2% of all estates.

2. Only a tiny fraction of family farms and businesses are affected

The Tax Policy Center estimates just 80family-owned businesses and farms will have to pay any estate tax in 2017.

The Congressional Research Service projects a higher number for any given year -- 159 -- but notes they represent less than 3% of taxable estates.

What's more, estate tax law offers options to significantly lower the value of a family business or farm for estate tax purposes.

Beyond that, "with a small amount of tax planning and for a relatively small amount of money you can avoid the estate tax," said Howard Gleckman, a senior fellow at the Tax Policy Center.

For instance, he said, parents running a family business can take out a particular kind of life insurance policy equal to the estimated amount of their estate's tax liability, and the proceeds from that policy can cover the tax bill when they die.

3. Some money in big estates is never taxed

Critics often portray the estate tax as a way to take a second or even third tax bite out of the apple.

That is, you made the money, paid income or investment tax on it while you were alive and then it's counted as part of your taxable estate, so is subject to tax again.

That definitely can happen, but usually only if you do zero estate planning.

But even in that instance, there's still a built-in allowance in estate tax law that ensures some money you leave your heirs will never be taxed. It's called a step-up in basis.

Say you bought shares in a company eons ago for $50 a pop. You never sell but bequeath them to your children.

When you die, the shares are trading at $150, for a gain of $100 per share. That gain will be tax-free forever.

You never paid tax on that $100 gain in your lifetime. And neither will your kids. That's because they get a "step up" in their cost basis. That is,if and when they sell those shares, their cost basis will be treated as $150, the value of the stock when you died. So they will only owe tax on the capital gains accrued above that amount.

It's possible that if lawmakers repeal the estate tax they may also repeal the step-up rule, at least partly. In fact, President Trump at one point proposed doing both during his campaign for estates worth more than $5 million.