Usually, a reference to stashing money under the mattress or in a shoebox is a joke. Real adults who make smart choices keep their money in the bank.

Or, at least, they should.

A new survey of more than 1,800 people from the American Express Spending and Savings Tracker found that 43 percent of Americans keep their savings in cash. An alarming 53 percent of those cash-hoarders “plan to hide bills in a secret location at home.”

While the survey itself doesn’t explain why, it’s reasonable to assume that at least some of these savers feel safer with money where they can see it, as opposed to hidden away in a bank that has played the bad guy since 2008. According to a 2014 Harris Poll, half of Americans say their trust in banks has declined in recent years.

Unless you have extenuating circumstances (although the only ones that come to mind immediately are running a drug cartel or operating as an international spy), avoiding the bank is usually a bad idea.

There are two primary reasons why:

It isn’t safe: Keeping your money in literal, tangible cash makes it extremely vulnerable. You could be robbed or there could be a flood, a fire, or a pest infestation that snacks on your dollars. Your toddler could find it and use it for art projects, your teenager could siphon it for Friday night expenses, or someone working in your home could find it just too tempting to leave alone.

Note, however, that products like mutual funds and life insurance are not protected. Then again, mutual funds and life insurance don’t exist in the shoebox in your closet.

It isn’t growing: Money in your pocket—or dresser drawer, or safe under your desk—loses value as it sits there. Even the less than 1 percent interest it would earn in a standard savings account is better than the 0 percent you’d get by keeping your money at home.

Inflation alone historically rises about 3 percent a year, and although there are various options to keep your money ahead of the curve (like investing in even the most conservative mutual funds or investing your retirement savings), keeping your money under the mattress is the quickest way for your wealth to essentially go backward.

Barring a flood, fire, or sticky-fingered visitor, you might have your savings in 30 years … but you won’t be able to buy as much with them as you would today.

You could also argue that storing savings in cash around the house could lead to miscalculations or tax complications (for instance, deciding to put more than $10,000 of cash into the bank could raise red flags) and could complicate payments (what major purchases does anyone make with a duffel bag full of cash, outside of the movies?).

The only feasible situation in which someone might want to keep his savings close at hand is if he has virtually nothing saved and needs that cash for everyday transactions until his next paycheck. If you have bigger savings goals, however, you’ll probably want to empty out the shoebox and head to the bank.