Expo Experiment Stories 2

This is the second article in a series of articles around interesting experiments we’ve run on Expo at Walmart Labs and how we analyzed results. Note that actual metrics have been changed and do not represent actual site performance, except to indicate directionality.

Guest Cart for Online Grocery

Overview

Guest Cart is guided by a simple premise: the lower the friction a user encounters while building a basket, the higher their propensity to place an order. In other words, the further users are down the ordering funnel, the more committed they are, and the more likely they are to put up with the friction of having to sign in or create an account and less likely to drop-off.

Guest Cart Round 1 — an A/B Test

Before the Guest Cart feature was rolled out to entire population, the users who came to Online Grocery were forced to sign in whenever they tried to: a) Add something to their cart, b) Change their store. Backed by the above hypothesis, we launched the first iteration of Guest Cart in Q1 2018 to move the signing in requirement down to the checkout:

Control:Need to sign in before they tried adding anything to their cart or changing their store

Variant:Free to perform the above two actions without the friction of signing in, sign in only when the user initiated checkout

However, the Guest Cart Round 1 failed to deliver the expected results. The major reasons unearthed for the poor performance of the feature were:

The landing page once the user initiated checkout: Instead of being funneled into the checkout flow, the users were landing on the homepage or any other page that they initiated checkout from after signing in, which led to high drop-offs.

No clear messaging regarding what happens to items when users have both Signed-In Cart and Guest Cart

The Merged Cart Scenario was faced by the qualified population on Desktop and the qualified population on Mobile.

During the course of this test we uncovered that the users who encountered merge scenario had lower probability of placing an order after initiating checkout compared to the users who did not see merged cart messaging.

On further analysis, we discovered that the most of users who faced Merged Cart scenario were removing items from their cart or changing their store after initiating checkout which added a lot of friction to placing an order thereby impacting our overall conversion metrics.

One positive thing that came out of this iteration was that the number of orders placed by users who created new accounts went up slightly.

Guest Cart Round 3 — Learn, Iterate, Succeed

Guest Cart Round 2 proved that merging the Guest Cart with the Signed-In Cart may not be the best possible solution. So, we performed one more iteration of Guest Cart in Q3 by introducing a third variant. This new variant will not merge the items in Guest Cart and Signed-In Cart. We called it the SWEEP SCENARIO.

In case of a user having items in both Guest Cart and Signed-In Cart, the Guest Cart will override the items in the Signed-In Cart and the user’s checkout store will remain the one that he had in Guest Cart.

After 2 weeks of testing, Variant 2 proved to be a clear winner and resulted in an increase in both Add to Carts and Checkout Initiations per visitor.

The increased checkout initiations converted to a lift in Placed GMV, Bought Items and Placed Orders per visitor for the qualified population of the test.

Continue Iterating, Continue Testing

Though Guest Cart Round 3 positively impacted all of our business metrics, we believe that we can still improve the user experience. There might be some proportion of population that prefers the merged experience over the sweep experience.

To address their situation, we are hypothesizing a Guest Cart Round 4 geared at further improving the user experience in a way where users who sign in will not have to lose the items which were previously in their Signed-In Cart. Instead, the Signed-In Cart items will still be accessible to the user in the even the Guest Cart Sweep Scenario occurs. We are optimistic that this iteration will also bring a better customer experience for our Online Grocery users.

This iterative testing allows us to apply the learning from previous tests and make incremental improvements to the user experience. Without A/B testing, we would not be able to measure the impact of the changes as accurately.