My most recent books are the Leader's Guide to Radical Management (2010), The Leader's Guide to Storytelling (2nd ed, 2011) and The Secret Language of Leadership (2007). I consult with organizations around the world on leadership, innovation, management and business narrative. At the World Bank, I held many management positions, including director of knowledge management (1996-2000). I am currently a director of the Scrum Alliance, an Amazon Affiliate and a fellow of the Lean Software Society. You can follow me on Twitter at @stevedenning. My website is at www.stevedenning.com.

Lean Startups Part 3: Most Changes Make Products Worse

In Part 1, I discussed some of Eric Ries’s ideas, including the mental shift from “build, measure, learn” to “learn, measure, build”, the “magic test” and the two leaps of faith.

In Part 2, I reviewed finding the right idea to test, learning from pissed-off customers, the three engines of growth, and the difference between user testing and market research.

In this article (Part 3), I turn to Ries’s provocative declaration that “most changes make products worse.”

Innovation is often seen as adding valuable new features to a product. The underlying assumption is that what the innovator sees as improvements will be seen as improvements by the customer.

Ries disagrees:

In Lean Startup, we take those experiments directly into the heart of product development and we say, “Look, if you’re actually building a product and delivering it to customers, wouldn’t you like to know if the features you’re adding are actually making the product better or worse?” Most people know in their heart that the features they’re adding are making the product better, but I beg to differ. My experience working with hundreds of startups is that most changes make most products worse. So we’re trying to take those techniques and really bring them into our everyday work in building a product.

He gives a striking example from one firm where the three-step registration process of a website was redesigned.

The original registration process was an obvious opportunity for improvement, with poor page design and “crappy radio buttons”. So it was decided to improve it with big, bold, beautiful three-dimensional pictures. When a split test was done, the new improved screens did 20% worse than the old one, which was devastating for the design team. What to do?

Ries explains:

The first impulse of most designers when that they get that kind of data is, “Well, it just looks so much better, let’s ship it anyway.”

I was like, “Are you kidding, ship it? No, no, no.”

The reason you don’t ship it anyway is not because we care so much about squeezing every last ounce of performance out of our registration process. It’s because when we fail, we have the opportunity to learn, if we take advantage of the opportunity.

Ries believed that the new registration design actually was better. But if it had a lower conversion rate, that meant that there was something about it that had been lost. There was something about the customers that they didn’t understand. So they decided to find out by doing split tests on each of the three pages of the registration process.

Ries says:

It was so simple it’s amazing that we didn’t see it. We found that the first two pages were awesome and had 20% improvement in the registration process, in getting people from page to page. But then we were giving up all those gains and by losing 40% of people on the third page, the download page, which was actually the simplest page. It was a page with this huge download button, with beautiful chrome and 3D specular highlights, just to make sure they didn’t miss it. Once we had that negative result, then we knew the right questions to ask.

Once we knew there’s a download problem, our tests got smarter. We’d bring customers in and we’d watch them go through the process. instead of asking them, “Do you love this page?” we watched what they did.

We noticed this funny thing. We would be like, “Download the product,” and they would pause and they’d be looking around the page. We’re looking at our watches, like, “Dude. Okay. Click the download!”

Finally we would get frustrated, and we’d be like, “So why aren’t you clicking the download button?” The person would say, “’I can’t find the download button.”

We’re like, “Why not? What about this giant thing?”

It was so big and obvious that users couldn’t see it!

When we fixed it, all the numbers went up. But the insight was more than just best size and color for the download button. The insight was, “Oh, we think about how we use our computer different than this person, and we need to be aware of that in our design.” So that was actually a very important moment there.

Work isn’t done until it is validated by the customer

The definition of what it means to say that work is “done” is a matter of intense discussion in the Scrum and Agile world, as reflected in an illuminating article by Mayank Gupta. A common definition would include completed code, unit tests completed, integration tests completed, performance tested and approved by the client or the customer proxy (the product owner ).

What Ries is suggesting is that it’s not enough that the client or customer proxy has approved the work in terms of agreeing that the feature conforms to what was asked for. In addition, you need to do a scientific experiment to determine whether the feature actually works better with real users in the real world. Is the product really better as a result?

Stop adding features & get rid of the crappy stuff

Continually adding features may in fact lead to a customer–driven death spiral, in which the product becomes increasingly unusable by the people in the real world. Although the product may in some objective or technical sense be getting better with more capabilities, it is actually getting worse in terms of the customer experience.

This underlines the need for a shift in focus from pushing more stuff at customers (inside-out focus) to an outside-in perspective of understanding the customers world and making the experience better for them.

“Editing also leads to great product designs and effective communications. According to Steve Jobs, ‘People think focus means saying yes to the thing you’ve got to focus on. But that’s not what it means at all. It means saying no to the hundred other good ideas that there are. You have to pick carefully. I’m actually as proud of the things we haven’t done as the things I have done. Innovation is saying ‘no’ to 1,000 things.’”

Imagine how much better our lives would be if all firms “got rid of the crappy stuff”.

What Eric Ries is doing for us in lean startups is giving us a methodology for accomplishing this on a systematic, scientific basis.

Post Your Comment

Post Your Reply

Forbes writers have the ability to call out member comments they find particularly interesting. Called-out comments are highlighted across the Forbes network. You'll be notified if your comment is called out.

Comments

Great article Steve! I just wanted to add, I’ve seen too much test marketing kill a successful idea. SOMETIMES just creating something and shooting from the hip is the way to go. These kind of products still contain that raw edge that gets refined out of things that are “Too processed” Its like correcting an artist for exact correctness. Their expression or juice is taken right out of their work. That same thing happens to alot of good ideas that are shot to death during their birth. The solution? Do a sample launch of both the rough version and the refined product as well. You will find people will often find charm in the original item “and then add their 2 cents worth” ultimatly making it theirs.

Point 1: “Work isn’t done until it is validated by the customer”, agree, but if I may, I would say, “Work isn’t done until it is validated by customers”.

Point 2: “Stop adding features & get rid of the crappy stuff”, agree to a large extent in the sense that least liked/used features must go and adding substantial capability without clustering UI.

Point 3: “In Part 2, I reviewed finding the right idea to test”, respectfully disagree on “finding an idea”, this is like you want to be entrepreneur to be an entrepreneur, not sure if that paradigm works out well.

Point 3: not sure I fully understand your comment. I believe that everyone in the organization must be focused on finding new ways to delight the customer. In that sense, for the firm to succeed on a sustained basis, everyone in the firm must develop a customer-focused entrepreneur-like mindset.

I paid $32.67 for a XBOX 360 and my mom got a 17 inch Toshiba laptop for $94.83 being delivered to our house tomorrow by FedEX. I will never again pay expensive retail prices at stores. I even sold a 46 inch HDTV to my boss for $650 and it only cost me $52.78 to get. Here is the website we using to get all this stuff, BidsNew.com

With regard to your thought, “In that sense, for the firm to succeed on a sustained basis, everyone in the firm must develop a customer-focused entrepreneur-like mindset.”, I appreciate your profound insight, entrepreneurship, as a mindset for an enterprise, you’ve transcended it to a higher level.