Q: Appreciate your thoughts on qualitative vs. quantitative approaches to market exploration. Customer development is like qualitative research; it is good for exploring hypotheses (and associated issues) but cannot validate them (by its very nature it is small sample size). To validate startup hypotheses one needs to do quantitative research, e.g. surveys, measurements (A/B tests), and have an appropriate sample size, statistically speaking.

I am a big believer in scientific method entrepreneurship, but you have to consider the different kinds of scientific models that are actually needed: physics, biology, zoology, human psychology, etc…Any successful product introduction requires a mix.

Scientific Method Entrepreneurship for Market Exploration

If we could eliminate the human element then markets become much easier to predict and much more amenable to techniques used in chemistry and physics to great success the last few centuries. But a startup that treats markets made of people as a collection of molecules at equilibrium in a gas is unlikely to prosper.

Botany and Biology tend to be more descriptive and less predictive because of the vitality of the subject matter. Antibiotics “proven” to work thirty years ago (or even last year) lose their efficacy in the face of adaption by bacterial population. Staph is still genetically staph, but some samples found in hospitals–and increasing not in hospitals–is no longer bothered even by methicillin. This caused pharmaceutical firms to overestimate the market for existing antibiotics and underestimate the need for new antibiotics and new techniques for rotating their deployment.

Zoology requires that we account for consciousness, making animal behavior even less predictable. Sparrow populations learn to sip the cream from the top of milk bottles left on suburban doorsteps and then to peck through the tin foil covers applied to prevent them from sticking their beak in. Each iteration of the milkman’s product was satisfactory only for a while, and that’s before we consider changes in human preference.

The study of human psychology is complicated even further by the presence of will on top of consciousness and vitality.

Careful Observation Guides the Crafting of Falsifiable Hypotheses

Physics, chemistry, botany, zoology, psychology are all sciences: they each rely on careful observation and falsifiable hypotheses. But to long for the certainly of a chemical reaction in planning a new product may blind you to entire categories of opportunity. You can be disappointed by a survey because you have shackled the interaction to the point where it’s impossible to be surprised–people cannot really tell you anything you have not already anticipated.

Most Useful Results Are Unanticipated

Most of the really interesting experimental results are unanticipated: “The most exciting phrase to hear in science, the one that heralds new discoveries, is not ‘Eureka!’ but ‘That’s funny …” (an insight attributed to Isaac Asimov). It’s not A or B but the realization that you should be asking a different question.

Heraclitus said that you cannot step into the same river twice, because the river is not the same, and neither are you. Markets evolve unpredictably but with increasing rapidity if we judge by adoption cycles for pervasive consumer goods (e.g. telephone, radio, television, microwave, cellphone,…). So a “statistically significant proof’ of solution-problem fit or product-market fit may have a shorter and shorter half-life.

I accept the Cluetrain Manifesto’s first thesis that “markets are conversations.” If you do then you are forced to reconsider where the real challenges are with new product development and introduction. If you are trying to join a real conversation there are only small sample sizes: your success may in fact be predicated on continually evolving your offerings to customize it to the particular needs of each prospect (or enable them to re-configure and customize it).

My hypothesis is that startups that organize and execute around discovery driven sales, mass customization of offerings, and genuine human interaction with their customers will substantially outperform those predicated on a commodity business model (high volume, low variation, low service). I make s strong distinction between measurement that observes and correlates actual customer behavior and their use of your product with automated testing methods that seek to substitute a mechanical interaction with prospects for a conversation.