How Etsy taught style to an algorithm

From “romantic” to “rustic,” the marketplace for handcrafted goods that express distinctive aesthetics is teaching its search engine to know what’s what.

Harry McCrackenJuly 11th 2019 8:00 amJuly 11th 2019

Building great search features for an e-commerce site is never simple. If it were, the results we get would feature a lot more irresistible gems—and fewer items we’d never buy.

At Etsy, the search challenge is particularly tough. The site’s stock in trade is not the sort of mass-produced goods that can be neatly categorized. Instead, 75% of the 60 million items that its 2 million merchants offer are handmade and therefore one of a kind. Even if they speak deeply to a shopper, they may do so for reasons that are difficult to divine from search terms and the information in product listings.

“We don’t have merchandisers entering the descriptions of the blue shirts in the pallets in the warehouse,” says Mike Fisher, Etsy’s CTO. That means that applying standard search technology to the wealth of products on the site produces results that are . . . well, generic. “If we put a bag in, it shows us different bags,” Fisher says. “Bags, bags, bags.”

Without Etsy’s new technology, a search for “large canvas bag” finds lots of large canvas bags, but doesn’t narrow things down by style. [Screenshot: courtesy of Etsy]

As it considered how to improve its search, Etsy concluded that it would be ideal to show shoppers items that not only matched their search terms but appealed to their particular aesthetic preferences. That would increase the chances that they’d love what they saw, which would be a boon for buyers and sellers alike. But doing so would require that the company teach its search engine to understand style.

After about a year of work, Fisher says, Etsy has trained a machine-learning model to effectively suss out the styles of items on the site, based on both textual and visual cues. The company is about to start testing results based on this new algorithm on the Etsy site. But it also believes that the technology it’s developed could have applications well beyond making e-commerce more relevant. Which is why three of Etsy’s data scientists have written a paper about it—coauthored with a Twitter employee—which they’ll present at the Association for Computing Machinery’s KDD (Knowledge Discovery and Data Mining) conference in August.

“We’ve open-sourced over 60 projects,” says Fisher of Etsy’s desire to share what it’s learned. “We’re big believers that we take a lot from the community, and we want to give a lot back.”

43 ways to express yourself

When Etsy’s technologists concluded that they needed to quantify style, they didn’t have to start from scratch. Staffers in the company’s merchandising group “spend their days understanding what’s being sold, what people are buying, what people are making on the site,” says Fisher. That team had quantified 43 different styles of goods that had traction on the site, such as “romantic,” “geometric,” “celestial,” “minimal,” “tropical,” “midcentury,” “rustic,” and “boho.” (“It’s really 42, but the 43rd is “everything else,” Fisher clarifies.)

Etsy’s merchandisers had even curated a list of 130,000 product listings that represented one style or another. That was plenty for their purposes, but a pittance when it came to training an algorithm. Using the 43 styles as a framework, Fisher’s group assembled 3 million data points from a month’s worth of search queries to feed into its machine learning model.

Many AI challenges can be well addressed with an off-the-shelf ML model. But Etsy’s wasn’t one of them. Since sellers don’t reliably convey a product’s style in their descriptions, scanning text alone produced results that were “okay but not great,” says Fisher. Likewise, standard approaches to image recognition couldn’t do the job on their own, since style isn’t solely about color patterns or other visuals that are easy for an algorithm to pick up on.

Etsy found its solution in a model that blended text analysis with an approach to image recognition based on its 43 styles. That let the algorithm draw conclusions such as identifying a piece of art depicting a whale as being “nautical” in theme—regardless of whether its listing used that term. And that opened up the possibility of directing a shopper to additional nautically themed products, even if they don’t involve whales.

One reassuring sign that the algorithm is classifying products correctly: When Etsy graphed the sales of items whose styles it had identified, they made sense. Stuff that the model identified as being “tropical” peaked in the summer months. “Romantic” goods did particularly well around Valentine’s Day and other holidays associated with gift-giving. “This gives us a strong belief that not only are we accurate with our styles, but they really resonate with buyers,” says Fisher.

“We think this is just the beginning”

Until now, Etsy’s quest to understand style has mostly been about creating, fine-tuning, and testing its algorithm. Now the company is ready to start putting it to work. Soon, it will test search results that emphasize styles based on the last 10 searches a given user performed, giving each person a custom set of products even if they search on the same term.

That’s just one straightforward way the style algorithm, in its initial incarnation, could improve product discovery. “We think this is just the beginning,” says Fisher. “Is 43 the right number of style categories? Who knows?”

Etsy should also be able to apply learnings from the algorithm to other aspects of the site besides search. In tests, Fisher’s team gave listed items their own “straight-up style score,” expressing the intensity with which they expressed a specific style. Logically enough, products that exude a particular look—really celestial rather than sorta celestial—sell better than ones that make less of a statement. That data could lend itself to useful merchant tools: “We think about this world in which we can help sellers determine whether they have a strong style or not,” says Fisher.

Even before Etsy gets to exploring all of its style-identifying technology’s possibilities, it’s learned some lessons. For one thing, even its big data isn’t that big when it comes to satisfying machine learning’s voracious appetite. “We collect a billion events a day,” says Fisher. “It sounds like a massive amount of data, but when you apply that to 60 million items, it actually becomes fairly sparse.” (One thing that’s helped: The company has spent the last two years migrating its service from its own servers to Google Cloud—a process that is now 75% complete—which means that it can keep a vast backlog of data on hand without having to manage everything itself.)

Spending so much time thinking about how to apply computer science to understanding the visceral feel evoked by products also reinforced something the company knew all along: The emotional attachment that shoppers form with an item based on its aesthetics are core to the company’s mission, “keeping commerce human.”

“On Etsy, style really matters, because there’s so much self-identity,” Fisher says. “But I don’t think we quite knew that it was so important to people.”