Accountability: You have to measure results. People on the growth team don’t always have to be successful. They don’t have to hit 100% success on experiments. But, the expectation is that they improve over time in terms of their knowledge of customer, product, and channels. Building a stronger base of knowledge means you’ll have more successful experiments.

How to Set Goals

Using this framework, ask yourself a question to form your objective. You should examine the one thing you can achieve that will drive the biggest impact on the growth curve. This requires you to take a step back and examine things before you dive into experiments. Once you have this, it forms your qualitative statement.

After the statement, comes a time frame, which is 30-90 days. Anything shorter than 30 days means you’re not being aggressive enough. Over 90 days and you’re biting off more than you can chew. It needs to be something you can make reasonable progress on in 30-90 days.

Once you have your objective and time frame, set three key results. These are quantitative measurements that indicate if you’re achieving your objective. The results are ordered by difficulty. Once you figure all this out, it is time to get to work.

Start by creating four documents. Their names and functions are as follows:

1. Backlog

This is where you dump all your ideas. This doc is a spreadsheet, and contains the experiment name, status, category (area you’re trying to improve), metric, prediction, and a resource estimate of how much time it will take marketing, engineering, design, and anybody else.

Anybody on the team can contribute ideas to the backlog. It should be a public document within the company. This document allows them to empty their head space and focus on the idea they’re currently executing.

2. Pipeline

This is the list of experiments past, present, and future. All past experiments have their results documented. This allows new team members a chance to go back and look at every experiment to show how they got to where they are today.

3. Experiment Docs

This is the most important document out of the four. Every experiment gets one of these. This document forces the team to think through the important elements of the experiment. When going through this document, they have to think about why they’re doing this experiment vs the others, what they expect from the experiment, how they’re going to design and implement the experiment, and record the learnings.

4. Playbooks

If an experiment is successful, they try to figure out ways to systematize them. They’re step-by-step guides for things they want to repeat.

Starting to get a little confused at the process and all the docs? That’s okay. Head over here and have a look at some sample docs and a Trello board that outlines the process.

A Breakdown of Each Phase

There’s a simple cycle for creating learning experiments.

Brainstorm > Prioritize > Test > Implement > Analyze > Systemize

Let’s elaborate on each phase in the process.

Brainstorm

In this phase, focus on brainstorming on the inputs, not the outputs.

Let’s say your OKR is set on improving an activation rate. Don’t sit there and try to figure out how to improve activation rate. There are probably thousands of ways to improve it, and focusing on this makes it difficult to come up with growth ideas. Instead, focus on breaking it down into very small pieces.

If the activation rate had three steps, you’d break down each and brainstorm ideas around each step. By keeping it focused on each step it becomes easier to come up with specific ideas about how to improve the inputs, which leads to an improved output. The book, Innovator’s Solution, reveals four ways to generate growth ideas:

1. Observe: Look at how others are doing it, both in your competitive and non-competitive space. If you have the goal of optimizing your referral program, look at other companies referral programs. Every member

Every member of your growth team should select a few programs they like, and the team can walk through each program. You’ll develop a ton of new ideas with this process. When seeing what other companies do, take it as inspiration, not a prescription. Take inspiration from them, and figure out how to apply it to your product and audience.

2. Question: A common exercise in this category is called “question brainstorming”. This is an hour-long meeting that consists of nothing but questions. Team members write questions on notes, announce the question, and put it on the board.

They ask as many questions as possible during this time period. This does two things. First, it helps reveal things they don’t know. Second, good answers start with good questions. The questions allow members of the team to start digging for answers. As they learn the answers to the questions, a ton of ideas on how to play off those answers

As they learn the answers to the questions, a ton of ideas on how to play off those answers tend to pop out of this process.

3. Associate: There’s a technique called smashing, where you take what you’re trying to improve and smash it with something completely unrelated. You learn from other processes and try to apply it to your experiment.

4. Network: Find a network of good growth people. Go to conferences, attend meetups, chat with others on the phone, etc. Exchange ideas with them – what you’ve been working on, what did or didn’t work, what you learned, etc. You’ll get ideas from others too, and will be able to feed it back into the process at your company. Once again, you get inspiration from others, not a prescription.
Once you have gone through this process, you create the backlog document.

Prioritize

This is where you prioritize what to work on first in the backlog.

Before we (humans) dive into a new idea, we tend to overestimate the probability of success. We inflate the impact of a success. And we underestimate the amount of time it’s going to take to test and implement. Because of these realities, it’s important to be brutally honest with three elements.

The probability that it will be successful, the impact it will have if it is successful, and the resources required to test and implement.

Probability is low, medium, and high. A high probability is usually an iteration that comes from a previous learning on an older experiment. If it’s low probability, it’s something that is new which no one on the team knows anything about. Impact is the most important one.

Every experiment idea has a hypothesis that looks something like this:

If successful [Variable] will increase by [impact], because [assumptions]

This format is great because it forces you to think through how it will work.

Test

Once it’s all prioritized, it’s time to create a Minimum Viable Test. It’s the minimum thing you can do to get data around your hypothesis. Outline the test in the experiment doc.

Implement

Not much to talk about here; just get the implementation done as quickly as possible.

Analyze

Once a test is done, it’s time to analyze. This is the most important step.

During your analysis, you want to pay attention to:

i, Impact: Results of the experiment

ii, Accuracy: How close to your hypothesis

iii, Why: the most important question you can ask. How did you see the results that you did?

Answering the ‘why’ is key. Why did it succeed, or why did it fail? Why were we close or way off from our hypothesis? Digging into why will help you understand things about your customer, channel, and product. This will lead to iterations and new ideas of experiments you should run next. Skipping this question means you’re blindly running experiments; you’re not taking what you learned into the next experiment.

Systemize

Once you find experiments that are successful, productize them with technology and engineering. If you can’t do it with technology and engineering, build into playbooks. As you hire and scale the team, you can point them to the playbooks so they know what’s going on and can repeat with minimal effort.