Starting on Sunday, Jan. 27, retailers will be allowed to tack a surcharge of up to 4% onto your tab if you want to pay with a credit card. If that sounds like a lot, you’re right. Happily, though, it’s unlikely to happen very often.

For years, card issuers have been making lots of money off so-called interchange fees. Until financial regulators and lawyers dragged this obscure term into general discourse, most people had no idea what an interchange fee is. (Still don’t? It’s the fee, typically about 2%, that a store pays your bank when you use a credit card at checkout.) For low-margin businesses like supermarkets as well as mom-and-pop stores that don’t have the clout of their big-box brethren to negotiate lower rates, these fees cut into profits in a big way.

But in a contentious legal ruling that is still being disputed, a U.S. District Court determined last year that merchants are allowed to pass along the cost of those credit-card interchange fees to customers. Consumer advocates say permitting surcharges is a slippery slope. “If a national sales tax of 2, 3, or 4 percent were being proposed, everyone would be up in arms,” ConsumerWorld.org founder Edgar Dworksy points out on his site.

But take a deep breath. You can avoid the fee by using a debit card, for one thing. And there are laws prohibiting these surcharges in 10 states: California, Colorado, Connecticut, Florida, Kansas, Maine, Massachusetts, New York, Oklahoma and Texas.

Ultimately, though, consumer awareness will be the strongest deterrent against widespread credit-card surcharges. Stores have to let you know with a sign on the door if they’re going to add a surcharge — although they don’t have to tell you how much it is until point of sale, when you’re already at the cash register. And today’s retail landscape is hypercompetitive, so many stores will be hesitant to risk alienating customers by charging extra for using plastic.

As consumers, we don’t take kindly to paying fees for stuff that used to be free. Bank of America and other financial titans found this out the hard way in 2011, when they tried to impose fees on people for using debit cards and were forced to backpedal in the face of customer ire. “Customer feedback in New Zealand and Canada confirms that consumers are not willing to pay more for the same products or offerings,” an article about credit surcharges on American Express’s small-business Open Forum warns. In both studies the article cites, customers overwhelmingly rejected merchants that implemented surcharges.

If consumers respond in a similar fashion here — and there’s reason to believe they would — surcharges probably aren’t going to become common. There could be exceptions in markets where there’s little competition; we’ve seen airlines get away with piling on fees, for instance, because there are relatively few carriers out there for all the people who need to get from Point A to Point B. Consumer advocates also say online retail could become a hotbed of surcharges because these merchants only have to disclose a surcharge when they ask about payment method — a step that’s usually near the end of the checkout process. Advocates worry that shoppers will be too invested at that point to be dissuaded by the charge and will just suck it up.

Another possible outcome if stores start passing on interchange charges is that consumers might reduce their use of credit cards. That too would probably trigger a rollback of surcharges: stores and banks, after all, know that consumers spend more when they pay with credit cards instead of cash — up to twice as much, by some estimates. And if we do use cards, we’d probably use more bank-account or prepaid debit cards, which aren’t subject to the surcharge. In other words, we’d pay less in interest, which is good news for us but bad for banks. Earlier this month, banking expert Dennis Moroney told TIME that credit cards are a big profit center for banks. They’re not going to let that revenue stream plummet over what’s essentially a 2% or 3% annoyance tax.

In a new online guide to credit-card surcharges, watchdog group Consumer Action points out that different cards will have different surcharges, so customers will have to keep track of not only which stores charge them but also how much using each card would set them back. This sounds like a giant headache, but it’s something that could ultimately wind up benefiting customers. If surcharges become the norm, expect credit-card promotions touting low- or no-surcharge transactions to become part of banks’ marketing mix, similar to what we’ve seen with the gradual rollback of foreign transaction fees on many new card offers. No, it’s not as exciting as a rewards program that lets you earn cash back or a free flight somewhere, but those points, miles or what have you were never really free. They came at the expense of retailers and — by extension — you and every other customer who indirectly pays for those fees through the prices retailers set for their goods.