Stores like American Apparel and Abercrombie & Fitch have gotten bad press for allegedly discriminating in their hiring by only choosing hot employees. But a new study shows that good-looking shoppers can be problematic for retailers, too. Self-esteem and shopping go hand-in-hand. To paraphrase a quote from former NFL-er Kevin Mawae, if you look good, you feel good, and if you feel good, you shop good. But three marketing professors found that women with low self-esteem can feel much less like buying if they see an attractive shopper trying on clothing.

“If she’s just holding the item, it’s not a big deal. If she’s wearing it and you’re not trying it on, it’s not a big deal,” says Darren Dahl, a marketing professor at the University of British Columbia. But if the same piece of clothing is being tried on by both women, that can be a deal-breaker.

In one experiment, the results of which will appear in the February 2012 issue of the Journal of Consumer Research, researchers asked female participants to go into a retail store and try on a certain shirt, under the guise of mystery shopping. An employee inside, in cahoots with the researchers, showed some participants an attractive shopper already wearing the same shirt, as if by coincidence.

After trying on the shirt, the shoppers filled out surveys evaluating the clothing and their own body esteem. Those with low self-esteem rated the product significantly lower if they had encountered the pretty shopper wearing the same item. “You just won’t buy it because the difference really magnifies,” Dahl says.

They found that women with low self-esteem felt a similar dislike for a dress if they tried it on after seeing a “consumer model” wearing it in an ad. (Consumer models are women featured in ads who are supposed to be “real” people who shop at the store rather than professional beauties.) They also found that the effect was less pronounced when a salesperson was wearing the clothing, as opposed to another shopper.

“The salespeople are put in a bit of a different category, if they work there and it’s their job to look a specific way,” Dahl says. There is less of a “people like me” effect driving the reaction. Still, the researchers speculate that retailers might think twice about forcing employees to wear only the store brand.

Women with high self-esteem meanwhile rated the products the same, whether they had seen the same piece of clothing on another shopper, the salesperson or a consumer model.

The researchers note that people are generally driven by a desire for self-evaluation, that we’re always looking for points of comparison that shape our self-images. Because people with low self-esteem are often made to feel worse by such comparisons, they will try to protect themselves by avoiding them. And when they can’t, that can amount to a lost sale.

Based on their findings, the researchers make some recommendations for retailers: Avoid dressing rooms that force shoppers to leave their stalls and stand in front of a public mirror. Be careful when using consumer models, because they may turn consumers off more than out-of-their-league professionals. Take it easy with the giant, in-store ads that feature models wearing the clothing, and save those for billboards.

“Whenever you’re creating the opportunity for comparison, that’s when you have to be a little careful,” Dahl says. “You can’t treat every consumer the same. … We all come into the shopping environment with different baggage.”

Katy Steinmetz is a reporter at TIME. Find her on Twitter at @KatySteinmetz. You can also continue the discussion on TIME’s Facebook page and on Twitter at @TIME.