Visualizing your experimentation backlog

Managing your experiment backlog

One thing that has helped me tremendously is physically visualizing my experimentation backlog to help me prioritize the team’s work.

Here’s the general principle I abide by when I allocate resources towards multiple options. I first consider the potential Expected Value (EV) of doing something. In a simple definition:

Expected Value = Probability x Potential Upside

Plotting your Expected Value

Every experiment or work item has a potential EV. It would be useful to plot out your experiments in a 2 x 2 matrix. The x-axis being Probability of Success and the y-axis being Potential Return. The matrix looks something like this:

When you have plotted everything out, what you’ll find is the following:

Top right corner (High-Impact, High-Confidence)

These are efforts that you’re highly bullish about. Not out of pure speculation, but because your past successes have given you confidence. Show these details in an Experiment Doc.

You should be able to speak to what past experiments have led you to this conclusion

These are the efforts you should focus on scaling up and putting in more resources on (these are where larger development costs start to make sense).

Top left corner (High-Impact, Low-Confidence)

Think about the quickest and fastest ways to test these things so that you can prove/disprove the hypotheses related to these experiments.

Once these experiments have been proven out, you will be able to move them to the top-right quadrant as your confidence increase

Bottom Left Corner (Low-Impact, Low-Confidence)

Deprioritize these experiments until you get qualitative or quantatitative data that challenges (and increases) the potential impact of these experiments.

Bottom Right Corner (Low-Impact, High-Confidence)

Don’t ever pick these things up – you’re confident that they don’t work!

Final note: Your return-on-investment depends on your cost

Add in lightweight t-shirt sizing estimates for each item. Within your top-right corner, you should think about which of these items you should prioritize relative to the cost.

Remember,

ROI = Impact(EV) / Cost

So make sure you have the highest EV to Cost ratio in the work that you do!