Why You Should Test Your App On Drunk People

Not long ago, Dave Lieb was sitting in his office, watching a man test Lieb’s contact-sharing app, Bump. Lieb had put out an ad on Craigslist, and people had trickled in to test the app in exchange for a small sum. Lieb watched the tester as he carefully read through every page of the app. At that moment, Lieb realized he had been going about user testing all wrong. "I thought, I bet it’s not like that in the real world," he recalls. In the real world, people aren't paid to use your app—would they really take such care studying it in such detail?

So Lieb started to take an unorthodox course of action: taking his app out into Bay Area bars.

A decade or two ago, maybe it made sense to do user testing in the office; most computing happened on a desktop, in a quiet environment. But in our mobile world, computing happens everywhere, under all conditions. "Our users could be anybody out at a train station, on the street, listening to music and texting a friend while trying to use the app," Lieb tells Fast Company. "Out in the real world, you’re bombarded by all these distractions."

Dave Lieb

Hence Bump’s series of boozy field trips. A bar is more representative of the environment in which a mass market app operates (and Bump is decidedly mass market, having been downloaded over 130 million times since its launch over four years ago). "Bars are crowded, people are moving out of the way or spilling beer on you," Lieb says. "There’s loud blaring music, people around you are cheering on a sports team, it’s often very dark." And if they’ve had a few drinks, people are a little cognitively depleted or impaired—not unlike the typical multitasker. "Drunk people are maybe a good approximation of distracted people," is the way Lieb puts it.

Lieb made it a requirement for all employees to help conduct one such field test at a local watering hole. And the trips proved worth their weight in gold, upending assumptions the team had about the app.

For instance: Lieb and company felt that an update of Bump had introduced a very clever mechanism: a feature that selectively enabled the "bumping" across of contact information from phone to phone. The basic idea was that certain pages of the app sent your phone into an active, bump-ready mode, while other pages were bump-disabled.

"That distinction, which seemed so clever in our minds, was totally lost on other people" at the bars, says Lieb. "They weren’t able to take the time to learn our distinction between the two pages." People just wanted to open the app and get bumpin’; they didn’t want to have to learn the categories of bumpable and non-bumpable pages. "People don’t have patience to read our elaborate marketing copy," says Lieb. So the team scrapped it, and in the next iteration of the app, Bump 3.0, every page was bump-enabled.

The team also learned that if you aspire to be a mass-market app, you have to be highly tolerant of user error. On one of Lieb’s visits, he sat down with a pair of women and asked them to use the app on their phones. They downloaded it, and when the app requested permission to use their location, they reflexively tapped "no." Lieb scrambled to explain that they ought to have said "yes"—since it could help the app work better—and he tried to show them how to go under the hood into their iPhone’s settings to give the app permission.

Another lesson from the bars: brand strength doesn’t necessarily correlate to usage.

"It was just a mess," he says, remembering the awkward fumbling that followed. "You leave an interaction like that thinking, ‘Man, we really failed, we suck, why is this app so bad?’" Fundamentally, the team realized that you "have to design an app to be tolerant of any action the user may want to take. If the user instinctively taps ‘no' to location sharing, you still have to make the app work."

Another lesson from the bars: brand strength doesn’t necessarily correlate to usage. One of Lieb’s colleagues went to a bar, where he came across a bachelorette party. He introduced himself as a designer at the app Bump, and the women erupted in cheers. The designer was psyched: die-hard fans! But when he asked, none of the women actually had Bump on their phones. "That told us that we had this brand," says Lieb. "We were an app that everyone knows about, but we had to figure out features they want to use" in their daily lives. At that point, Bump pivoted, beginning to expand into an all-purpose file sharing app. With the new Bump, users can share all types of files, and they can even pass files simply back and forth between their phone and laptop.

So drinkers are helpful testers. Lieb further recommends testing apps with the very young and the very old, and he urges the importance of getting a sense of how users use the app outside of the Bay Area, where "if you grab a random guy off the street, probably the chances he’s a software developer are greater than 20%." The Bay Area—any tech-centric area—can be an echo chamber reinforcing the feeling that your mediocre or esoteric ideas are actually brilliant and ready for the mainstream. Lieb's team corrects for this bias by conducting phone interviews with users from all over the world.

It’s the bar visits, though, that really helped Lieb and company steer a better course. "For any app that aspires to be mass market, I think bars are a really good place," he says. "They’re always full, the people there are always happy, and they’re always willing to talk to you for a beer."

Add New Comment

4Comments

Clever man. He has realised he can get paid to sit in bars drinking whilst doing his job.

On a serious note though, this approach has some merit, but I'm not convinced that drunk people always have the same decision making behaviour as sober people. So I would have to look at the specific context to assess whether bar user testing is reliable. Also recording for later analysis is going to be difficult.

You are right to assume that a lab in an office isn't always the best place for mobile based user testing. But don't forget people often do use their phones at home or somewhere more relaxed. It would be interesting to see a study that looked into the differences in interaction behaviour in different environments.

Great article. Last year, as a bit of a farce, I started a"company" Three Sheets Market Research, that does usability testingon drunk people. While the methodology was unorthodox, and the goal was humormore than strategy, I was pleasantly surprised at the abundance of insights weactually witnessed. Even more interesting was the reaction from UX designersand researchers. Those who weren't appalled (thinking it was a seriousendeavor) remarked that testing on drunk people was a valid, and underutilized,approach.

Of course, in this article's example, the test goes a necessarystep further by testing the app in the field.

That's really smart actually. I was just thinking about running a usability test for the Government of Canada mobile website versions but couldn't figure out how to test normal Canadians. Doing it at the mall or on the street corner is okay but it's not really somewhere where people are willing to stop and chat - they want to go places. But at a bar, people are there to drink and to pick-up but doing something a little odd, people will want to sit with you.