Think about it: A tax refund is just that — a refund of your own hard-earned money. It’s not a gift or a stroke of good fortune. The problem is that most people don’t look at tax refunds this way.

Most Americans—a full 75%—receive refunds after filing their taxes. In other words, most Americans have too much money withheld from their paychecks. More than half of Americans—58%, to be exact—say they intentionally plan to receive a refund each year. Understandably, people do so to avoid an unexpected tax payment come April 15, with the idea that it’s better to withhold a bit more to be on the safe side. But the average tax refund will be over $2,800 this year — well beyond “a bit more.”

Clearly people have other motives for over withholding, or they just don’t put much thought into it. Here are the four ways we’re irrational when it comes to tax refunds, and arguments for why it’s better to approach the topic far more logically:

1. Mistaking Uncle Sam for Your Rich Uncle

Wouldn’t it be great to have a wealthy relative who sends you a fat check every spring? That’s how many view their tax refunds: as gifts from the government. On Facebook, Trish, 36, posted a photo of her new juicer along with a caption that read, “Thanks Uncle Sam!”

In the course of my research, I spoke with many consumers like Trish, who seem to think that tax refunds pretty much appear out of the blue. Max, 55, told me that he knows he’ll get back at least $1,000 every year. He generously earmarks $1,000 for his church, but anything above that is what Max calls “free money.”

When people view their tax refund as a gift or some kind of reward or bonus, they spend it differently. They’re more likely to splurge, and it’s easier to rationalize purchases. For example, Nancy, 48, normally prides herself on her frugality, but when it comes to her tax refund she admits she’s developed a habit that’s out of character. “I always seem to get my refund when I’m yearning for warm weather,” she says. “Three years in a row now I’ve bought full-priced spring shoes. I never buy full-priced shoes. I don’t know, I just feel like it’s a bonus and I treat myself.”

Tax refunds are often called interest-free loans to the government. That’s exactly what they are. Lately, interest rates are low to non-existent, so you’re not missing out much in the way of interest by having more money than is necessary deducted from your paycheck. Still, you are missing out. Those who over-withhold are putting slightly less money in that college or retirement fund when they let the government keep their money, interest free, throughout the year.

And remember, if that extra money isn’t saved throughout the year and is instead handed over in a lump sum, there’s a higher likelihood that you might just spend it. According to the National Retail Federation, 12% of those receiving tax refunds will spend it on a vacation, and 13% on a major purchase such as a car or television. Meanwhile, 42% say they’ll save at least part of their refund.

As illogical as it is, some people say the act of depositing a large sum of money into a savings account makes them happier than saving little by little, paycheck after paycheck. “I love making a really big deposit,” says Cleo, 39. “Last year I got back almost $3,000, and I don’t know if it would have been so satisfying to put in a few hundred dollars a month.”

Some people know themselves well enough to assume that they wouldn’t be able to save throughout the year. “I’ve thought about changing my taxes, but I really count on my refund for our vacation,” says Sandra, a 40-something mom of three, who is skeptical of her ability to save. “It always seems like something comes up.” One solution is to automatically have a portion of paychecks set aside into a special savings account. That would seem to be more sensible than letting the U.S. government hang onto your money for most of the year.

3. Paying Interest on Your Own Money

Debbie, 33, says she feels a huge sense of relief when her tax refund arrives, not so she can afford a vacation or a new TV—but so that she can pay off the credit card bills from holiday purchases. “I hate seeing that I’m paying all those finance charges, and I always file my taxes as soon as possible so that I can get the refund,” she said.

I asked Debbie why she didn’t withhold less so that she could avoid finance charges altogether. “I might spend it,” she replied. “And then I wouldn’t be able to pay off the Christmas bills.” Debbie doesn’t normally carry a balance on her credit card, and she estimates that she spent about $90 in finance charges just as a result of holiday purchases.

Debbie’s not alone in her approach. Nearly 42% of those receiving a tax refund plan to use them to pay down debts.

But if you wait for a tax refund to help you pay off debts, how much extra are you paying in the meantime? Someone with a $2,800 balance on their credit card will pay nearly $500 in interest fees by making the minimum payment each month for a year on a card with an 18% interest rate. On the other hand if that person had less of his paycheck withheld so that he had an extra $235 per month to add to his minimum payment, he would be debt free in 10 months and pay less than $240 in interest.

4. Letting Inertia Take Over

For most, the idea of rejiggering withholdings is a chore. It’s not just that the paperwork is tedious. There’s also the added anxiety that you’ll do something wrong and wind up owing a huge amount at tax time—or worse, wind up audited.

All of which explains why most people don’t bother. “Believe me, I know I should change my withholdings,” said 40-something Alexander, who received a refund of over $5,000 last year. “I just didn’t get to it. I’m going to do it for sure this year … probably.”

If you’re feeling ambitious, or just curious, the IRS has a calculator tool to help determine how much you should be withholding. And FYI, you can change your withholdings at any time during the year.

Kit Yarrow chairs the psychology department of Golden Gate University and was named the university’s 2012 Outstanding Scholar for her research in consumer behavior. She is a co-author of Gen BuYand is a frequent speaker on topics related to consumer psychology and Generation Y