At least, that's according to Ron Friedman, a partner at Marcum LLP, an accounting firm, and a long-time consultant for clothing manufacturers, who spoke with TheStreet on Monday, July 30.

Friedman believes with the current upbeat economy shoppers are heading to full-price retailers rather than discounters such as TJX, the industry leader, or Ross Stores, because they have more money to spend.

"The economy is good. People are going out to restaurants, clubs, movies and they're shopping at the Nordstrom's and Macys of the world," said Friedman. If tariffs kick in and retailers need to pass them on to consumers, he said, it may send them into the arms of off-price retailers for bargains.

However, analyst Simeon A. Siegel, of Nomura Instinet Equity Research, told TheStreet on Monday, "That [any exodus to cheaper stores] will last about a week, before full-price retailers start discounting."

He takes a long view of tariffs in that they are one of the many elements, like the costs of labor, cotton and other materials, and freight, that retailers and manufacturers must circumvent to make their operations more efficient. As for what determines whether consumers face higher prices, he said, generally depends on the pricing power of retailers and manufacturers.

Jon Gold, vice president, supply chain and customs policy of the National Retail Federation, said all retailers and customers will feel the pinch from higher prices if tariffs, which so far have sidestepped broad categories of clothing and footwear, are put in place. "As the President further escalates tariffs against China" he added, "those categories will most certainly be on the list and subject to a tariff."

As for the off-price retailers, Siegel thinks off-price retailers have nothing to fear because their business is up and will continue. Year to date, TJX is up 26%, and Ross, up 7.6%.