Validated Learning

Video transcript

Eric Ries — "When I was an entrepreneur I kept having this very unpleasant experience. I would work really hard on a product, tinker with it, get it all as ready as I could possibly make it. Then right before it was time to launch I always felt sick. Is this product really good enough to show to people? What if it crashes their computer? What if it erases their hard-drive? What if they don’t like it and they tell their friends? I used to have this image in my mind that some enterprising journalist would notice it on launch day and write this huge expose: ‘Idiot Eric Ries Doesn’t Know What Quality Means - Never Hire Him Again’.

And there’s my mugshot like in a prison movie. Terrible, terrible, nightmares. But most of the time when we would launch a product none of those bad things would happen because no one would even try the product. No one would even find out how bad it was, and so on the one hand my first reaction was relief - at least no one found out how bad my design decisions were and I didn’t crash any computers. And then I would think wait a minute - why did I just work for months or years of my life on a product that nobody is using? That’s terrible.

That’s the first thing that we have to realize when we become entrepreneurs: most of the time we are building the wrong things, and we’re doing it really efficiently. If you read something like the Agile Manifesto it’s right there that our unit of progress is not waterfall style, you know - specifications and documents. We’re going to make our unit of progress a line of working code. Every bit that we add to the product - that’s progress. Everything else is waste.

That was pretty good for a while, but in the era of entrepreneurship that’s not good enough. Because if we are building the wrong product really efficiently, it’s like we are driving our car off a cliff and bragging about our awesome gas mileage. We have to also be testing to figure out are we are actually going in the right direction. When I started to understand this concept I began to notice this very strange phenomenon - which is that: every day I would try to get a certain number of customers to use my product. In the early days of my last company we spent five dollars a day - that was our whole marketing budget. In those days Google ad cost five cents per click so that meant we could buy one hundred clicks per day. That means that every single day one hundred human beings are coming to our site and we can measure how that cohort of customers behaves. Of the hundred that come to our site every day, how many buy our product?

If you have ever actually done this experiment you very quickly become a conspiracy theorist, because basically it will be the same every day. We went months where every single day one in every hundred customers would buy our product. I started to be like ‘Wait, what the hell?‘ Are today’s customers on a conference call with yesterday’s customers? "Hey guys, what’s up? How many have you bought? Okay, one?‘ It was so consistent, and yet in the language of science these were time independent controlled trials. Each day’s customers didn’t talk to yesterday’s, they were a reflection of how good our product was.

Now here’s the amazing thing. Over the course of several months we were making the product better every day - fixing bugs, adding features, improving usability, improving design, and yet every day the report card was coming back the same: C- c-. c-. c-. I started to call into question - are we actually making the product better? Or is it possible that all the work we’ve done over these months is a waste? It turns out - it’s the latter. It’s a waste! If we are not affecting the way customers behave then our improvements aren’t improvements at all.

In the lean start-up we say that the unit of progress is something that we call ‘Validated Learning’. Learning is a dangerous concept because you can’t deposit learning in the bank, you can’t return learning to your investors, they aren’t excited about all your learning. Every Joe Schmo can tell a good story about how much they learned, about how their failures were a good learning experience. That doesn’t count. What we need to do is prove (scientifically) that we are learning how to build a sustainable business. Whatever our theory is, in how our business is going to be successful - we need to show, cohort by cohort, that those behaviors that we think are so important are actually going to happen. Sometimes it’s really straightforward, like in the case of my last company where we were looking at what percentage of customers buy our product. Very straightforward. The conversion percentage needs to be going up over time or we are not actually making the product better. But that doesn’t have to be the key behaviour that you measure. If you are trying to do a viral product like Facebook, that will travel like an epidemic through the population that also makes very specific demands on how customers should behave.

For example: the thing people forget about a virus is that it is not optional. If Ebola is ravaging my town I can’t be like ‘Well I opt out! I don’t want the word of mouth.‘. This is not word of mouth, this is a virus - an epidemic. An involuntary spread of the product. If I show a product like Facebook to even ten customers - say every day I showed a product like Facebook to ten customers a day, day in and day out - the theory of viral growth makes very specific claims about what should happen, namely: those ten people should infect ten more people, who infect ten more people - the epidemic should be out of control. If that’s not happening it calls into question whether our plan is even going to work.

Now, the key takeaway of the concept of validated learning is to treat everything we do as entrepreneurs as an experiment - as a scientific experiment designed to help up figure out if we are actually on the path to a sustainable business."