'Made in the USA' is a tough promise for luxury brands to make

President-elect Donald Trump looks on as LVMH Chairman and Chief Executive Officer Bernard Arnault speaks to reporters after their meeting at Trump Tower in New York, U.S., January 9, 2017.

Standing in the lobby of New York City's Trump Tower last month, LVMH CEO Bernard Arnault told reporters that the Louis Vuitton brand will expand its manufacturing in the U.S., potentially setting up a factory in the Carolinas or Texas.

Yet even as President Donald Trump pressures other businesses to ramp up their American-based production, experts agree that the "Made in the USA" movement is unlikely to take hold in the luxury sector anytime soon.

Designer labels face several major headwinds when it comes to producing goods here, including inadequate infrastructure and a shortage of raw materials.

"You've got so many things working against you," Edward Hertzman, an apparel supply chain expert and CEO of Sourcing Journal, told CNBC.

Among the challenges luxury brands would facde include finding factories to fill their orders. The number of apparel manufacturing establishments in the U.S. dropped 58.2 percent from 2000 to 2014, to 6,891, according to the latest data from the U.S. Census Bureau.

Even if they got production up and running, these labels would still need to look outside the U.S. for their raw materials. Hermes, for example, would not be able to source the crocodile skin used for its Birkin bags inside the U.S., said Hana Ben-Shabat, a partner in the retail practice of A.T. Kearney. Meanwhile, China and India are the largest producers of silk.

Labor is another headwind luxury brands would encounter — though for a different reason than what's usually cited. Because true luxury items are often crafted in high-cost countries like France or Italy, producing them in the U.S. would likely require a similar wage for workers.

However, the bigger issue is there are not enough trained seamstresses or craftsmen in the U.S. to fill demand, Hertzman said. Even if there were, stitching luxury items here could take away some of the products' allure, Ben-Shabat said.

"What does it do to the credibility of the brand?" she posed. "People [may have been] buying the brand because it was French and it comes with that history."

It would make more sense for accessible luxury brands with American roots to move some of their production to the U.S., as they can leverage their domestic heritage as a selling point, Ben-Shabat said. Shinola is one example of a brand that prides itself on its Detroit roots and American-built watches.

The growing popularity of the "see now, wear now" model could also shift some production back to the U.S. Designer Rebecca Minkoff previously told CNBC that her brand has leaned on local production in New York City and Los Angeles to speed up the process of getting designs from the sketching room to the selling floor. The brand would eventually like to make all of its products in the U.S., Minkoff said at the time.

Still, these types of initiatives are unlikely to move the needle in a significant way, Rick Helfenbein, president and CEO of the American Apparel & Footwear Association, told CNBC. Only about 3 percent of apparel sold in the U.S. today is manufactured here, compared with about 46 percent two decades ago, according to AAFA's data. Although that number has ticked slightly higher over the past three years, it's not statistically significant, Helfenbein said.

"We speak to a lot of our members and they're all rethinking how they do business," he said about his trade association's 1,000-plus members. "That doesn't necessarily mean that you'll see a tremendous increase in U.S. factories doing this, but there will be an attempt to use more U.S. fabric to run pilot plants [here] to see if things can be delivered quicker."

A Republican proposal that threatens to slap a 20 percent tax on imports sold in the U.S. has also sparked conversation about bringing apparel manufacturing back to America. Yet even if the legislation passes, it will take time to set up the infrastructure required to bring jobs back to the U.S. In the meantime, it threatens to wipe out companies' earnings and put them out of business, Helfenbein said.

"It's meant to favor exporters and disfavor importers," said Helfenbein, whose group opposes the measure. "That's great but when you're in an industry that's 98 percent imported it's a serious threat."