A developer in London

The Gmail bug that’s been stealing $187M a year from Expedia

Disclaimer: The events depicted in this story are purely factual and described to the best of my memory.

Prelude

Many years ago, in the City of London…

The Expedia empire reigns. Having conquered the world of travel.

Its two heads, Expedia and Hotels.com share the wealth in roughly equal measure, a few tens of billions each every year, for holding a tight grip and a large commission on all planes and hotels ever used in the world.

It is a cold day of winter in London, where the Hotels headquarters is located. A perfectly regular business meeting is about to take place, as every last Tuesday of month.

Chapter 1 – Free buffet

Teams and leaders will give numerous presentations about the business, products, partners, competitions and many more topics. Attendance is mandatory for all new employees and contractors.

Some talks prove to be interesting. One would learn why are there constant harassment popups like “Over 9000 people have booked this hotel in the last 24 hours.” and whether there actually were 9000 people who booked rooms.

One of the presenters gather people and improvise a usability test at some point. Think of a place you’d like to go for holidays, search it and go through the booking, as if you were intending to go.

Unbeknownst to all, one of the most trivial yet impactful bug in history is about to be discovered.

Chapter 2 – What Could Go Wrong?

I’ll spare you the full narrative of the session. Except, one insignificant detail toward the end, one person in the back of the room saying she cannot login to the site.

If a decade of experience has taught me anything, it’s that for every user who complains of an issue, there could be another million who don’t, in spite of the issue being very real and painful.

Always investigate feedback, no matter how stupid or irrelevant it could be, it could be the tree that hides the forest.

For reference a small tree made of pure gold would be worth a few hundreds million dollars

Chapter 3 – Debugging

Editor’s Note: The scene was originally happening on mobile. Screenshots were captured on desktop for illustration.

What can be so special that it doesn’t work for her? Let’s find out.

She cannot login because she forgot her password. She did the “reset password” many times and it does NOT work.

Why would this not work? Let’s have her try and reset it one more time.

Forgot Password => Reset => “Instructions have been sent to …”

The email is allegedly sent… and it is received by her Gmail.

A new message in the inbox! (desktop screenshot)

She opens it and click the reset link and it doesn’t work.

Why does this not work? The page says “invalid or expired link, try again to be sent a new email“.

So, let’s try again. Except this time, it’s my turn.

Reset the password again. Open Gmail open, it shows that there is a new unread message again. Great!

Open it. Looking carefully, there is something unusual at the bottom of it…

There are messages collapsed at the bottom, as in an old discussion (desktop screenshot)

The latest message is not shown upon opening. It’s showing an old message instead (in the middle of the page that was truncated for the screenshot). The newer messages are collapsed at the bottom. They have to be opened manually.

It turns out that Gmail hides new messages that look similar to previous messages.

The password reset messages are there, hidden at the bottom, each with its own unique link. We open the latest email manually and it works! The password is reset.

Debugging complete. It’s not a bug, it’s a feature.

She’s been opening the same old reset email every single time, with an obsolete link. That’s why reset didn’t work for her.

Gmail displays an old obsolete email when opening the discussion. No matter how many newer emails are received, they’re all systematically ignored and collapsed. She never noticed the hidden messages at the bottom.

Every reset message comes with a unique link, that gets invalidated when a new reset request is done, hence the errors about using an invalid link. She would never be able to reset her password without being able to view the latest email.

The collapsed messages are fairly hard to spot on desktop and almost impossible to spot on mobile. She doesn’t stand a chance.

Chapter 4 – Estimation of Impact

The password reset procedure is failing for a multi billion dollar website with hundreds of millions of customers.

Customers buy accommodation when they travel, more or less frequently. It’s reasonable to expect a sizeable user base to forget their password as often as every single time they return, especially if they travel sparingly.

The issue is impacting all users who forgot their password, use Gmail (not sure about other clients) and don’t notice the hidden messages being at the bottom (it’s really hard to spot).

Let’s estimate the impact. Incidentally, all the required metrics were distilled through today’s presentations and the data team just introduced themselves.

Assuming some percentages of some percentages of some statistics from users and sales. (read: private numbers).

The direct impact of this bug is a direct loss of revenues of $187M dollars per year, simply accounting for people who are unable to login and place any order. Existing users cannot use the service, new users can register but not return.

Then there is the indirect loss that is very hard to measure. If you were in the room, you would notice participants switch to the competition (booking.com) within a minute of frustration, especially if already familiar with it. It is brutal from a business perspective. Not sure if users might be giving up any faster or slower because of the artificial circumstances.

The full impact accounting for direct losses, plus indirect losses, plus recurring losses, plus reputation loss, plus competitors stealing our business, well, forever stealing our business because the site is unusable and lost users won’t switch back, etc… is hard to put an estimate on. It’s easily a multiple of the number above.

Chapter 5 – Fix

How to force Gmail to stop hiding similar emails.

Well, let’s make the email dissimilar. Adding the current time to the subject line ought to be enough.

The software is in Java, a verbose language. It takes 60 characters to format a date. An easy patch, easier done than said.

You knew the 10X developer, a developer who is as profitable as 10 of his peers. Allow me to introduce the unicorn developer, a developer who is worth a billion dollar by himself.

Conclusion

Actually, the bug is still active. I never had the opportunity to fix it, didn’t stay long.

During the hallway testing, I noticed at least 2 other people who couldn’t login and talked about it out loud. That’s a minimum of 3 out of 30 people impacted, proving with certainty that a major percentage of users were affected. Some guys actually said that there was a known issue with password reset and it was being worked on.

Long story short. It was not worked on at all. The bug is still there and it’s been there all along. I got it recently when I booked a travel.

It’s been going on for many years. It must be well over a billion dollar loss by now.

I’ve seen the email change a couple of times. Pretty sure it was rewritten from scratch, possibly more than once, yet the issue persists. I genuinely wonder if anyone ever investigated, let alone understood the root cause.

It is truly a global Gmail bug. I’ve noticed a handful of collapsed emails from various sources since then. The impact is not limited to Expedia.

11 thoughts on “The Gmail bug that’s been stealing $187M a year from Expedia”

Gmail always puts messages in the order they were received – that’s how you read them, from first to last. The optimal behavior would be to collapse previously read emails and expand unread emails – which is exactly the behavior I observe in Gmail.

Also, the person who “fixes” (breaks) this isn’t adding value to the internet; they are shifting it. People are still booking their flights or hotels, just maybe not where Expedia et al want them to.

This same logic supplies to your comment about there being a billion dollar loss. That would imply that only Expedia is affected, when in fact they also benefit from the behavior affecting other vendors. It could, in fact, be a wash.

At that scale, standard practice is to measure and track account recovery funnels. The customer service department would also flag the outreach by people that were impacted by this so-called bug.

I think this is a case of assuming that people booking trips online still don’t know how to use Gmail 2019 (you are wrong; they do) and attributing that to a random dollar figure. You have zero data to support any of these imaginary scenarios.

Doing a user testing session with 30 people and having 3 of them unable to login is strong data. Having one of them repeat the procedure 3 times before my eyes to demonstrate that it’s completely broken is undeniable evidence of an UX issue. As for the rest of the numbers, that’s internal.

Don’t ignore the pain of users just because there are some who can find their way through. Don’t think that people will call support to reset their account, they don’t, the support number is itself fairly well hidden 😀

Seriously? Doesn’t anyone see the problem root cause? Expedia and Hotels.com are loosing money because of their poor design of password recovery process. Not because of Gmail.

Making email titles unique is just a workaround, but not a solution.

Problem is that password reset link that is emailed to the user is re-created each and every time user clicks “Forgot password” button on the website. Even if previously sent link was not clicked, nor expired.

Indeed, Gmail has this feature that stacks unread emails into a thread, starting from oldest message, and keeps next emails collapsed. However, this should not impact user journey.

Correct user journey should be:

1) user clicks “Forgot password” button on the website
2) unique link is sent to him, that is valid for next 24h, or until it clicked.
3) if user clicks the link, he is immidiately logged into the website, that preserves his previous sessions (page, selected dates or product or whatsoever he was doing before he decided to log in); login link gets invalidated; also old password gets invalidated; user is shown a non-modal hint to set new password. He can do it at any time, even after he finishes his activity and reaches goal he had in mind when came to the website.
4) alternatively, if user attempts to click “Forgot password”, system checks whether password reset link exists, not used and not expired – in this case just sends same link again. No need to create new one.
5) profit!

Yes, even now, Sometimes “Gmail displays an old obsolete email when opening the discussion. No matter how many newer emails are received, they’re all systematically ignored and collapsed. So, you never noticed the hidden messages at the bottom.”