Enterprise UX: Enterprise Experimentation Wrap-up

Welcome to Fuzzy Math’s recap of the Enterprise UX 2015 conference. We’re organizing our blog posts around both the keynotes and the four conference themes: Insight at Scale, Craft amid Complexity, Enterprise Experimentation, and Designing Organizational Culture. Today we recap the first theme of day two, Enterprise Experimentation.

The first theme of day two, wrangled by Bill Scott, VP of business engineering at Paypal, Alissa Briggs, director of user experience at Brigade, and Jeff Gothelf, principal at Neo, was focused on how well, and sometimes how not-so-well, different approaches work at bringing experimentation into an enterprise. There was more overlap between these talks than there was with any of the other themes, which was indeed nice, and a pretty stark contrast from yesterday’s theme.

One of the more interesting points that each of the speakers touched upon was failure. Bill and Alissa made the point that you are going to fail, it’s guaranteed; however, how you fail, when you fail, and what you do with your failures are going to determine your long-term success. Bill mentioned that you need to “fail small and learn, don’t fail big on a huge bet,” while Alissa mentioned that you should “fail in small ways that won’t really matter at the end of the day,” and, “big fails always matter.” She also made the point that you should anticipate failure and when it does happen, look the data in the face, do some research to figure out why the failure occurred, and always plan the next experiment. You’re not finished with this experiment until the next one has been planned.

You aren’t done with this experiment until the next one is planned

There are two parts to what they said that I really liked. First, the idea that you aren’t done with this experiment until the next one is planned feels right to me. Here at Fuzzy Math we think of research as a thread throughout projects, not just a step at the beginning or end, and I like thinking of experimentation in the same way. Second, failure as part of moving forward in the design process is a concept that we strongly believe in; it’s even in our pitch deck, and it can sometimes feel weird explaining how we “fail” as part of our process, but it’s true. It was genuinely uplifting to hear that this is also part of the process at Paypal and Brigade.

Teams who come up with winning ideas get to continue the journey with them — they become the core team for the new product

Rather than highlight how failure is a required step in the design process, Jeff highlighted different ways that enterprises fail at innovation. To paraphrase, he basically said that incentivizing features doesn’t work, hackathons don’t work, and innovation labs don’t work, either. Jeff’s idea for what does work is an internal innovation studio. In order for this to work, it requires patience, buy-in from the C-suite, funding that is separate from the rest of the organization, and radical transparency around what goes on in the lab. Bill and Alissa enjoyed that, I’m sure, as Jeff made special note that both successes and failures need to be included in what’s made public. One of the bigger ideas in Jeff’s talk that stuck with me was that teams who come up with winning ideas get to continue the journey with them — they become the core team for the new product. “This is entrepreneurship for 40+ year-olds,” as Jeff said, and I buy it. Great talk.

Other interesting random bits from the theme:

“Treat the UI layer as the experimentation layer.” – Bill

“When engineering becomes about engineering, you have a problem. It should always be about the customer.” – Bill

“All buildings—or in this case software—are predictions, and all predictions are wrong.” – Bill

Intuit runs tens of thousands of experiments per year, leading to millions of dollars in revenue. – Alissa

Experiment coaches are on each team, with 10-20% of their time spent on experimenting. – Alissa

The visuals from Jeff’s deck were really fantastic – I highly recommend giving them a look in the deck