Still, consumers might not be willing to spring for really expensive underwear right now.

"You’re going to see a separation between the high-end makers and the more Costco-like makers. We’re a buyer of Costco for exactly this reason," he said, while shares of Hanes and the like "don't appeal at the current prices."