Lessons from Our Recent A/B Button Test

In the startup world a few fancy phrases get tossed around frequently; things like "lean product development," "a/b testing" and "iteration." They all basically boil down to one central idea, though: test as much as you can, look at the data and make changes based on that data.

At Chicory we do this on a daily basis. So, I thought it would be fun to walk you through a recent test of ours to see which "button" styles, if any, result in higher click through rates. Why does click through rate matter? Well, it means that people are engaging with our product, interested in shopping for the recipe and using grocery delivery. The more clicks a button variation receives, the more we can infer that that button produces better engagement. Simple.

The Test

In early January we launched a number of tests on our "flagship" product for food publishers, what we lovingly call the Chicory “Get Ingredients” button. We call it that, but in fact, we tested out a few different calls to action--"get," "buy," "order." All in all, we were running about 35 different variations of this button and looking at a simple metric to judge success: click through rate.

Looking at the below, you'll see the different variations and combinations that we tried out. Obviously, the CTA varies (order, get, buy), as does the shape of the button (rounded, "plain" meaning square), shadow vs. no shadow and "inverted" (the inverted button appears white, but fills in with the Chicory blue-green when someone hovers).

Another important factor in the testing is scale. Tests only matter if they’re statistically significant. There needs to be a real, clear difference in your variations. Our platform of over 450 recipe publishers--giving us 35 million monthly impressions and 15 million monthly unique users--allows us to get scale on these tests quickly.

The Results

Here are some of the results from our testing:

The “inverse” style buttons performed the best. In fact, the initial style that we use in decks and presentations (the "plain" button) was the worst performing--goes to show what a little bit of testing can tell you!

Returning to the CTA on the button, we discovered that we can thankfully still call this our “Get Ingredients” button--this CTA performed far better than any of the others, which was actually a little surprising since we thought ‘get’ would be much less interesting than "buy" or "order" to a potential grocery shopper.

The Takeaways

Custom Button Styles

We’ve seen a trend among our own platform partners (whether food bloggers, large publishers or brand publishers) for more custom buttons and custom integrations. While there’s a clear appeal of buttons looking integrated with a site, we consistently see these custom button variations as low performing.

While the button may not look like it’s a part of your brand, that’s okay. Chicory is a tool and we’re constantly testing to make sure consumers understand that and use it. We suggest that Chicory publishing partners opt-in for all button and other a/b tests that we’re performing.

Test Out Assumptions

Test out your initial touch points for consumers! One thing that our testing shows us is that assumptions about branding and style are often wrong. You might not be testing out button styles, so instead consider design elements that speak to your KPIs (ours was button clicks, yours is likely traffic).

Whether you’re a food blog or large food website, Google and Pinterest have made it so that your recipe photos are the most important initial touch point with readers. How can you optimize performance of your photos to result in more traffic? Test our your assumptions: are long pins really performing better for you; have you tried out a horizontal pin any time recently? Are people more likely to click your Facebook post if, in the photo, there are utensils present vs. no utensils?

We’re Not Even Close to Done Yet

This was some insight into one of the many, many tests that we perform on a daily basis at Chicory to make all our products better whether they’re for consumers, publishers, brands, or retailers.

Even the “Get Ingredients” button is far from fully tested--we haven’t even talked about different colors, animations or countless other testable factors.

There’s no “one size fits all” for grocery ecommerce. We’re all new players in the grocery ecommerce space and at Chicory we’ll be the first to admit that there’s a lot of experimentation to be done in this space. Beyond just “button style” there are more holistic experimentations that need to be done for both publishers and brands to truly enable a seamless ecommerce shopping experiment. Stay tuned to our learnings at blog.chicory.co for more information!