How to Test Your Product Ideas by SoulCycle VP of Product

In order to increase the certainty of a product, it is important to leverage different kinds of tools, each one is unique and tailored to different kinds of situations to achieve a wide range of goals. Which one is the right for you?

Meet Tim Holley

Tim Holley leads the Product Team at Soul Cycle which is a New York City-based fitness company with 88 studios in the US and Canada. He has a Masters in Management Science and Engineering from Stanford University. Before working at SoulCycle, Tim worked at Etsy where he made significant contributions to the company. He helped Etsy grow from $525M in GMV in 2011 to $3.2B in 2017 by delivering impactful products.

Existential Questions

Where are you going?

Think about product strategy, vision, and principles.

What does great look like?

Set a goal and work towards it.

Do you have a line of sight to achieving your goals?

This plays a very important role; don’t put all the eggs in one basket; use a portfolio approach.

For example, when you’re starting a project, the budget of the project and the certainty of success should be inversely proportional; for a $1M project, the certainty should be around 20%.

How do you increase certainty?

Pragmatic > Dogmatic: You have to be Pragmatic versus Dogmatic. In Etsy land, nothing went out without being A/B tested, they were so incredibly dogmatic that if you couldn’t A/B test it, it wasn’t really a thing. And that is probably and intellectually the right thing because you can really quantify the impact. But in reality, it slows you down and it’s potentially limiting because launching a new feature and A/B testing a new feature is really difficult.

A/B testing is not cheap and sometimes it’s not the answer. A/B testing is just not the silver bullet that delivers answers to you. But if not that then what? The answer is below.

Use a set of tools to increase certainty in different situations

For example, you can use the “Pilots/Betas” tool on a sample of 100+ people and can get feedback on product usability and understanding of the infrastructure.

You can use the “1:1 Interviews” tool on a sample of 20 people and get potential directions on where to head.