You probably know by now that you should speak with customers and test your idea before building a product. What you probably don’t know is that you might be making some of the most common mistakes when running your experiments.

Mistakes include testing the wrong aspect of your business, asking the wrong questions and neglecting to define a criterion for success. This article is your guide to designing quick, effective, low-cost experiments.

A Product With No Users After 180 Days

Four years ago, I had a great idea. What if there was a more visual way to learn about our world? Instead of typing search queries into a text field, what if we could share visual queries with our social network and get information about what we’re looking at? It could change the way people search! I went out, found my technical cofounder, and we started building a photo Q&A app. Being a designer, I naturally thought that the branding and user experience would make the product successful.

Six months later, after rigorous usability testing and refinement of the experience, we launched. Everyone flocked to the app and it blew up overnight. Just kidding. No one cared. It was that devastating moment after a great unveiling when all you hear are crickets chirping.

Confused and frustrated, I went back to the drawing board to determine why this was. My cofounder and I parted ways, and, left without technical expertise, I decided to step out and do some research by interviewing potential users of the app. After a few interviews, the root cause of the failed launched finally dawned on me. My beautifully designed solution did not solve a real human need. It took five days of interviews before I finally accepted this truth and slowly let go.

The good news for you is that you don’t need to go through the same pain and waste of time. I recently started working on another startup idea. This time, I followed a structured process to identify key risks and integrate customer feedback early on.

A Product With 16 Paying Customers After 24 Hours

I work with many entrepreneurs to help them build their companies, and they always ask me for feedback on the user experience of their Web and mobile apps. They express frustration with finding UX talent and want some quick advice on their products in the meantime. This happens so frequently that I decided to learn more about their difficulties and see what might solve their problems.

I specified what I was trying to learn by forming a hypothesis:

“Bootstrapped startup founders have trouble getting UX feedback because they have no reliable sources to turn to.”

To test this, I set a minimum criterion for what I would accept as validation to further explore this opportunity. I had enough confidence that this was a big problem to set a criterion as high as 6 out of 10. This means that 6 out of the 10 people I interviewed needed to indicate that this was a big enough problem in order for me to proceed.

By stating my beliefs up front, I held myself accountable to the results and reduced the influence of any retroactive biases. I knew that if these entrepreneurs already had reliable sources to turn to for UX feedback and that they were happy with them, then generating demand for an alternative solution would be much harder.

In three hours of interviews, I was able to validate the pain point and the need for a better alternative. (You can watch a video walkthrough of my findings.) My next step was to test the riskiest assumption related to the solution that would solve this problem. Would they pay for an online service to get feedback from UX designers?

Instead of building a functioning marketplace with designer portfolios and payment functionality, or even wireframing anything, I simply set up a landing page with a price tag in the call to action to test whether visitors were willing to pay. This is called a pitch experiment. You can use QuickMVP4 to set up this kind of test.

Behind the landing page, I attached a form to gather information on what they needed help with. Within a few hours, 10 people had paid for the service, asking for UX feedback on their websites. Having validated the demand, I needed to fulfill my promise by delivering the service to the people who had paid.

7Did not build any functionality; just a form to collect information. (Large preview8)

Because this market is two-sided — with entrepreneurs and designers — I tested the demand side first to see whether the solution provided enough value to elicit payment. Then, I tested the supply side to learn what kind of work designers were looking for and whether they were willing to consult virtually.

To my surprise, the UX designers I spoke with had established clientele and were very picky about new clients, let alone wanting to consult with random startups online. But they all mentioned that they had been eager to take on any work when they were first starting out and looking for clients. So, I switched my focus to UX designers who are not yet established and are open to honing their skills by giving feedback online.

Armed with these insights, I iterated on my landing page to accommodate both sides of the market and proceeded to fulfill the demands of the customers I had accumulated. No wireframes. No code. Just a landing page and two forms!

11A landing page that tests both sides of the market, simultaneously. (Large preview12)

13Interest in this service can also be measured by their willingness to fill out the form. (Large preview14)

To simulate the back-end functionality, I emailed the requests to the UX designers, who would then respond with their feedback, which I would email back to the startup founders. Each transaction would take five minutes, and I did this over and over again with each customer until I could no longer handle the demand.

Do things that don’t scale in order to acquire your earliest customers and to identify business risks that you might overlook when implementing the technical aspects. This is called a concierge experiment. Once the manual labor has scaled to its limit, then write the code to open the bottleneck. Through this process, I was able to collect feedback on use cases, user expectations and ideas for improvements. This focused approach allowed for more informed iterations in a shorter span of time, without getting lost in wireframing much of the application up front.

Today, BetterUX.it15 is a service through which startup founders connect with UX designers for feedback on their websites and apps, and designers get paid for their expertise and feedback.

How To Create A Product That People Want

What did I do differently? The structured process of testing my assumptions saved me time and confirmed that each part of my work was actually creating value for end users. Below is a breakdown of the steps I took, so that you can do the same.

Should You Build Your Idea?

My first mistake with my first startup was assuming that others had the same problem that I experienced. This is a common assumption that many gloss over. Build products that scratch your own itch, right? But many entrepreneurs realize too late that the problems they’re trying to solve are not painful enough to sustain a business.

As product people, we often have many ideas bubbling in our heads. Before getting too excited, test them and decide which one is the most viable to pursue.

Design An Effective Experiment

To get started, break down your idea into testable elements. An effective experiment must clearly define these four elements:

hypothesis,

riskiest assumption,

method,

minimum criterion for success.

At Lean Startup Machine, we’ve created a tool called an Experiment Board162, which enables us to easily turn crazy ideas into effective experiments in a few minutes. As you go along, use the board as a framework to design your experiment and track progress. Refer to the templates provided on the board to quickly formulate your hypothesis, riskiest assumption, method and success criterion. You can also watch my video tutorial for more information on designing effective experiments.

Construct a Hypothesis

Every experiment starts with a hypothesis. Start by forming a customer-problem hypothesis. Once it is validated, you can go on to form a problem-solution hypothesis.

Define your customer.
Which customer segment experiences the most pain? They are your early adopters, and you should target them first. These are the people who have the problem you’re solving for, and they know they have the problem and are trying to solve it themselves and are dying for a better way! Most people have trouble identifying these customers. If you do, too, then just segment your potential customer base by level of pain and differentiating characteristics, such as lifestyle and environmental factors. Being specific will reduce the time it takes to run through experiment cycles; once you’ve tested against that one segment and found that the problem doesn’t resonate with them, you can quickly pivot to test another customer segment. In the long run, having a clear idea of who you’re building for will help you maintain a laser focus on what to prioritize and what to dismiss as noise.

Define the problem.
What problem do you believe you are solving for? Phrase it from your customer’s perspective. Too often, people phrase this from the perspective of their own lofty vision (“The Web needs to be more human”) or from a business point of view (“Customers don’t use our service enough”). Also, avoid being too broad (“People don’t recycle”). These mistakes will make your hypothesis hard to test with a specific customer, and you’ll find yourself testing for a sentiment or an opinion in interviews, rather than a solvable problem. If you have trouble, phrase the problem as if your friend was describing it you.

Form a hypothesis.
Brainstorm on a few customers and problems to consider all the possibilities. Then, combine the customer and problem that you want to focus on with this sentence: “I believe customer x has a problem achieving goal y.” You have just formed a testable hypothesis!

Identify Your Riskiest Assumption

Now that you have formed a customer-problem hypothesis, poke some holes and extract the riskiest assumption to be tested. Start by brainstorming on a few core assumptions. These are the assumptions that are central to the viability of your hypothesis or business. Think of an assumption as the behavior, mentality or action that needs to be true in order to validate the hypothesis.

Ask your team members, boss or friends to suggest any assumptions that you may have overlooked. After listing a few, identify the riskiest one. This is the core assumption that you are most uncertain about and have the least amount of data on. Testing the riskiest assumption first will speed up the experiment cycle. If the riskiest assumption is invalidated, then the hypothesis will be invalid, and you will have saved your company from going down the wrong path.

Choose a Method

After identifying the most critical aspect of your idea to test, determine how to test it. You could conduct three kinds of experiments before getting into wireframes. It’s best to start by gathering information firsthand through exploration. But you could choose a different method depending on your level of certainty and the data.

ExplorationConduct qualitative interviews17 to verify and deepen your understanding of the problem. Even though you experience the problem yourself, you don’t know how big it is or who else has it. By conducting exploratory interviews first, you might realize that the opportunity isn’t as big as you had thought or that a bigger problem could be solved instead.

Pitch
Make sure the solution would actually provide value by selling the concept to customers before building the product. This will measure their level of determination to solve the problem for themselves. A potential customer not taking a certain action to use your service, like paying a small deposit or submitting an email address, indicates that the problem is not painful enough or that you haven’t found the right solution.

Concierge
Personally deliver the service to customers to test how satisfied they are with your solution. Did your value proposition meet their expectations? What was useful for them? What could have been done better? How likely are they to return or recommend the service to a friend? These are all insights you can discover in this step.

Set a Minimum Criterion for Success

Before running the experiment, decide up front what result will constitute success and what result will constitute failure. The minimum criterion for success is the weakest outcome you will accept to continue allocating resources and pursuing the solution. Factors like budget, opportunity cost, size of market, level of demand and business metrics all play into it.

The criterion is usually expressed as a fraction:

“I expect x number of people out of the y number of people in the experiment to exhibit behavior z.”

I like to set the criterion according to how big I think the problem is for that customer segment, and then determine how much revenue it would have to generate in order for me to keep working on it. At this point, statistical significance is not important; if your target customer segment is very specific, then testing with 10 of them is enough to start seeing a pattern.

Once you have validated the hypothesis with a small sample of the customer segment, then you can scale up the experiments to test with larger sample sizes or with other segments.

Run the Experiment

Once you have defined these elements, you are ready to run the experiment! Have team members look at your Experiment Board and confirm whether they agree with what you’re testing. This will hold you and the team accountable to the results, so that there are no subjective arguments afterwards.

Analyze the Results and Decide on Next Steps

After gathering data from your target customers, document the results and your learning on the Experiment Board. Did the results meet your criterion for success? If so, then your hypothesis was valid, and you can move forward to test the next risk with the product. If not, then you need to form a new hypothesis based on your learning to get closer to something that holds true. Track your progress over time on the Experiment Board to get a holistic picture of your validated learning and to continually make informed decisions.

Test and repeat. You’re on your way to creating a great product that people want!

Grace is the Co-Founder of Javelin, enterprise software for product teams to test the viability of new ideas. She has trained executives from GE, ESPN, and American Express, in the process of rapid experimentation for product validation and is a frequent speaker on Lean UX. As a designer and entrepreneur, she is constantly exploring the intersection of creative production and impactful creation. She writes about her experience at uxceo.com.

Matt Smith

“Within a few hours, 10 people had paid for the service, asking for UX feedback on their websites.”
The thing here is, you already had a base of people for your site. Some nobody with no contacts isn’t going to get 10 people, yet alone 1 person.

How do you test an MVP if you are a complete unknown in the field, and have zero contacts? I think those are the more valuable experiments for the majority of people.

Dave Sanders

I agree Matt. This solves about half the equation for product testing, but there is a huge gaping hole in the “how do you get those first 10 people to sign up – WITHOUT accidentally over-promoting.”

Its one thing to test a market this way, but the MAIN test is “can I get 500 people to even see this to get those 10 signups?” You still need to have effective marketing to get the concept out there to begin with. You might have the best idea ever and get 100 signups in 24 hours – but not if people don’t know it exists, leading to a false test of your hypothesis where the problem is the advertising – not the idea.

I could also see the scenario where, due to accidentally effective advertising your concierge service is overwhelmed. Granted, this is a nice problem to have but at the same time you don’t want to strangle your idea in the crib because you can’t perform the service. The last thing you want is to have a killer idea and then disappoint a lot of people while they wait 3-4 months to get the actual product out there.

It’s definitely an interesting process and I like the scientific method being applied this way, but its not a complete tool.

John

Apart from the obvious reasons, why are you not doing this yourself? I understand that some start-ups are targeting the start-up community and you see a little chance to do something that way rather than struggle yourself.

But I imagine that I’m not the only person who is suspicious of why people claiming to have so many great methods are not doing this themselves.

I have noticed that each thing like this always has a little history of how you struggled too, then came up with a nice method that you want to share with others that I imagine you know full well has a lot missing.

Grace Ng

That’s a good point Matt. An easy way to find your customers is to think of where they might hang out – meetup groups, online forums, LinkedIn groups, shopping mall etc. I believe there is no such thing as “no contact in the industry” anymore; put your social network to use or search for interest groups or conferences where your potential customers congregate.

Daniel Schwarz

For me, the defining moment when building my site was discovering Criticue. It uses a tic-for-tac system where you rate websites and they rate yours. I found it to be very enlightening, where the feedback was incredibly insightful and included things that I never would have thought of. Since then our engagement has been much higher and I now use it for almost everything that I make. Testing along the way solves major hiccups later on.

Grace Ng

Elco

Thanks for this post Grace, I found it very useful, it’s clearer for me now that I’m running experiments and I’m facing with some challenges along the way on how to do this or that, this post gives me a north to follow…

Colin Hutchinson

software validation

Every time I come to #hostname you have another interesting post up. A friend of mine was talking to me about this topic several weeks ago, so I think I’ll e-mail them the url here and see what they say.

software validation

am speechless. It is a unbelievable weblog and very partaking too. Great work! That’s probably not a lot coming from an beginner blogger like me, but it’s all I may assume after having fun with your posts. Nice grammar and vocabulary. Not like different blogs. You actually know what you are speaking about too. So much that you made me want to learn more. Your blog has turn into a stepping stone for me, my fellow blogger. Thank you for the detailed journey. I actually enjoyed the 6 posts that I have learned so far.

Tomek

Will

Great post, the scientific method applied to product development sounds like the right way to approach it.

I actually created a website (http://valid8or.com) that allows you to “pre-validate” your startup idea by trading feedback with other entrepreneurs, but I can’t stress the importance of user research as well.

Leave a Comment

Yay! You've decided to leave a comment. That's fantastic! Please keep in mind that comments are moderated and rel="nofollow" is in use. So, please do not use a spammy keyword or a domain as your name, or else it will be deleted. Let's have a personal and meaningful conversation instead. Thanks for dropping by!

Subscribe to our email newsletter for useful tips and valuable resources, sent out every second Tuesday.

Meet Smashing Book #5, our new book on real-life responsive design. With front-end techniques and patterns from actual projects, it's a playbook to master all the tricky facets and hurdles of responsive design. Save 25% today.

Fixing RWD issues can be quite easy — once you understand exactly why they come up. The Mobile Web Handbook will help you understand technical issues on mobile and how to deal with them effectively.

Hungry for more content? Over 60 eBooks are waiting to be discovered in our lovely Smashing Library. And guess what? You can watch Smashing Conference talks there, too.

SmashingConf isn't the eighth wonder of the world, but we are pretty close. Join us at at the shores of Santa Monica for SmashingConf LA on April 27–30 or at SmashingConf NYC on June 15–18. You won't be disappointed.