The assumption is that you have a great product idea and seek validation from customers before expending vast resources to build and bring it to market.

Indeed, it makes sense to transcend conventional approaches to making product decisions. Intuition, sales anecdotes, feature requests from customers, backward industry thinking, and spreadsheets don't form the basis for sound product decisions. Incorporating lean startup concepts, and a more scientific approach to learning markets, is undoubtedly a sounder approach.

Moreover, in larger organizations, sometimes further in the product life-cycle, everyone seems to have an opinion about such aspects of the business model as:

The most pervasive and urgent market problems the product should solve.

Whether the solution truly addresses the market problems.

What price customers would be willing to pay.

What tactics will most effectively drive prospects to buy.

Presumably, the "validated learning" approach of lean startup enables the organization to identify which opinions are factual and not mere speculation.

Validation in Practice

But let's consider what our naive lean startup practitioner (we'll call her "Cameron") does in practice.

Present the idea

Cameron puts together a slide deck, minimum viable product (MVP), or demo, presents her idea to one or more prospects, and eagerly awaits feedback. The prospects say, "I like it!" or "I would buy it!" Cameron feels warm inside, as the prospects have "validated" her and her idea.

Ask about the "pain point"

Since solving an urgent and pervasive problem or "pain point" is usually a prerequisite for product success, Cameron visits with prospects and asks them if they experience one of the problems she envisions her product would solve. "Yes," reply the prospects. Feeling "validated", Cameron does an internal high-five with herself.

Name a price

Cameron asks prospects what they would pay for a solution with the features she envisions for her product. Prospects "name their price". Or, in equally naive fashion, Cameron herself names a price and asks, "Would you pay X for the product?" Prospects reply affirmatively. Cameron can hardly contain her giddyness as her pricing assumptions are "validated".

Cameron feels extra special for having "gotten out of the building" to visit customers.

What's Naive about Cameron's Approach?

The results of all of Cameron's efforts are practically worthless, aside from the emotional affirmation she may feel. When you visit prospects, you don't get reliable information by posing hypothetical questions. When you seek validation from someone, you will tend to get it. Cameron unwittingly designed her interactions with prospects in a manner likely to confirm her preconceived ideas, and her interpretations of the results were naive. Let's call it "validation bias", an insidious psychological disorder that has infected the lean startup community.

Note: had Cameron conducted a survey, though seductive because it would have yielded quantitative data, the results would still have been suspect to the extent the questions were hypothetical and failed to confront prospects with real choices and commitments.

What's the Alternative?

If you want to be scientific in your approach to product decisions, you craft experiments and make falsifiable predictions, with the intention of testing, not "validating", the hypotheses and underlying assumptions.

Philosopher Karl Popper is famous for having championed falsificationism, a set of principles that shifted the emphasis from verifying scientific beliefs to ensuring they are possible to falsify via experiments:

According to Wikipedia:

"A statement is called falsifiable if it is possible to conceive an observation or an argument which proves the statement in question to be false."

In fact, Popper argued that a statement isn't scientifically factual at all if it isn't falsifiable.

Cameron could also work with her team to craft "digital" experiments and predict how those experiments will turn out. For example, her team could author "how-to" guides, downloadable on landing pages, that help prospects solve the problems she assumes they face. The team could predict how many people will visit those landing pages and how many people will proceed to download the guides. (To make prospects aware of the how-to guides, the team could run Facebook ads or Google AdWords to drive people to those landing pages and see if the predicted number of page visits and downloads materializes.)

The possibilities are endless, but they require a mindset that emphasizes falsifying product ideas and business model hypotheses, not "validating" them.

14 comments
:

Roger, I also like this post - but I'm a bit slow. It seems like most customers would provide "falsified" feedback on one or more features/core concepts. (After all, customers come in all shapes & sizes.) What am I missing?

Thanks, Sachin. We agree that lean startup methods aren't to blame. It's the implementation of those methods that can lead organizations astray.

However, neither of the sample questions you posed is likely to yield reliable information. While it's possible that the prospect will answer "no", they may be hesitant to let you down with a negative answer.

Here's a great YouTube video illustrating how "confirmation bias" is part of human nature, and we tend to craft our experiments to verify, rather than falsify, our hypotheses: http://www.youtube.com/watch?v=vKA4w2O61Xo.

Great post Roger, this is probably painful reading for many would-be entrepreneurs that all too often shift back into their emotive way of feeling rather than applying critical thinking. I'm always reminded of the stats, over 90% of SUs will fail to return initial capital, emotions are the reaons why.

After reading the LeanSU manual front to back it's clear to me how many entrepreneurs haven't bothered to read the materials. Those that have, how many take it as a cast iron blueprint to success whilst jumping on the hype bandwagon after attending a few seminars/workshop surrounded by positive high fives. Reminds me of the Tony Robbins self help crew.

I hear these terms like lean SU, validated learning, actionable metrics knowing that most have very little deep scientific understanding about implementation or why it's so vital.

The problem with many entrepreneurs is themselves which they never factor in their equation, hence the absolute need for rigorous scientific analysis.

As Elon Musk explains "natural human tendency is to suffer wishful thinking". If it is true that business is nothing more than an amplification of your mind then you better account for your imbalances, limited scope of thinking by applying rigorous scientific method.

I always use Elon Musk as an example because no matter what business he puts his mind to quite simply they work.

Why is someone like Elon able to deliver consistently over a few decades across multiple industries?

here's a trick. When conversing with someone as YOUR idea. Pretend it's from some anonymous third party committee who's neither an idiot nor a genius, keep it neutral. Then, the person you're speaking with will not worry about being polite to you (when seeking validation) or avoiding insulting you (when seeking falsification). As a facilitator of the conversation, always maintain that the designer is an anonymous committee, no individual; thus all those "nice" social behaviors will hopefully be less potent. Also frame the questions such that the "nice" participant is "speaking for the masses of users"; then they might *take advantage* of their empathy and speak more truthfully out of concern for the masses over just the designer.

This trick also gives you an out when they observe or criticize; it helps you avoid getting defensive or answering too quickly (which will give away this pretense). You can always hedge and redirect by saying, "I'm pretty sure it works like X, but what is natural to you" or "Hmm, I'm not sure, we'll have to ask them-- what do you think is best?"

Jaime, thanks for sharing your "trick". It does help to get more objective feedback on an idea if the audience doesn't think you'll take it personally.

However, there's still a problem. You can facilitate a conversation that uncovers problems people face and the current ways they are addressing the problems. But most people do not accurately predict what they will do in a hypothetical situation.

"Would you buy this product that this other person is developing?" will therefore not yield meaningful insights in most cases.

People are experts on their actual, current situations and challenges. And when confronted with new, real-world choices, they will act. But they do not know how they would act in a hypothetical future situation.

Subscribe To

About Me

I'm a downtown Austin dweller with a passion for food, football (playing it, not watching it), knowledge, nutrition, investing, and a low-car lifestyle. In my professional life, I empower teams to make smart product decisions by applying design thinking, lean startup methods, and timeless marketing principles.