Friday, March 14, 2014

Cohort Analysis: A (practical) Q&A [Guest Post]

My colleague Nicolas wrote a great guide with tips and tricks on how to do cohort analyses which I'd like to share with the readers of this blog. Thanks, Nicolas, for allowing me to guest publish it here. Without further ado, here it is!

- - - - - - - - - -

At Point Nine
we believe that the only way to get a real sense of user retention and
customer lifetime is doing a proper cohort analysis. Much has been said
and written about them and Christoph has a published a great template and guide on the topic if the concept is new to you.

With
this Q&A I want to focus on some of the more practical questions
that might arise when you are actually implementing a cohort analysis
for your startup. After close to two years of working with SaaS
companies and doing numerous of these analysis I have learned that in
most cases there is no perfect step-by-step procedure. But although you
will always have to do some customisation for a cohort analysis to
perfectly fit your business, there are a handful of questions and
pitfalls that I have seen over again and again and want to share so that
you can avoid them.

Now let's get into it!

Q: Which users should I include in the base number of the cohort?

There
are two parts to the answer as it depends on what you want to measure.
If you want to find out your overall user retention and have a free
plan, then you should include all signups of a specific month.

However
if you are trying to calculate your customer lifetime value, you should
only look at the number of paid conversions. I only count an account as
a paid one when the user has or will be charged for a period. So if you
offer a 30-day free trial for example, wait to see if the user converts
into a paying plan before you include him in the cohort. This way the
numbers won't be biased with users that actually never paid for your
service.

If possible without too much effort, you should also
try to eliminate all 'buddy plans' that you have given to friends, your
team or investors. If they are not paying, they are not representative
for the real cohorts.

Q: How do I treat churn within the first / base month?

There
are different approaches here, but in my view taking churn within the
first month into account is the most accurate representation of reality.
That means that in your first month the retention could be less than
100%, if people cancel their paid subscription within that month. It
would look something like this:

I
do this because I don't want the analysis to exaggerate churn in the
second month and understate it in the first / base month. After all the
reasons for churning in the first 1-4 weeks could be very different than
after 5-8 weeks.

Q: Should I treat team and individual accounts differently?

If
you are at a very early stage or sell mostly (90%+) individual plans it
is probably sufficient to mix them all in the same analysis. But when
team plans make up a significant part of your paid accounts, or your
product has a very different user experience when a whole team uses it,
you should probably look at both type of accounts separately.

Findings
could include that team accounts are a lot more active, churn less and
see a lower drop-off in the first month than individual plans. Or not.
:)

Q: What about annual vs. monthly plans?

Again,
if you are focusing on how active your users are over their lifetime it
is OK to mix both plans. If you just want to see how many of the people
that signed up still come back after X months, no need to split hairs.

If you are however focused on churn, you should only look at paid accounts that could have churned in that month. This is one of the 9 Worst Practices in SaaS Metrics
and means that you should exclude all annual plans that are not
expiring in the respective month. Including these in the denominator
would otherwise skew churn numbers.

Q: Now that I have it, what can I take away from it?

The
two most obvious take-aways are depicted in this (KISSmetrics) retention
grid. Note that this is a most likely an analysis for a mobile app and
the numbers for your SaaS solution should be significantly higher:

(click for larger version)

Moving horizontally you can see how the retention of a cohort decreases over the users lifetime. Interesting here is where the highest drop-offs occur and whether the numbers stabilise after a few months.

Vertically, you can (ideally) see how the retention of your cohorts change over the product lifetime.
Assuming you are not twiddling your thumbs while catching up with House
of Cards or sipping Mai Tai’s at the beach once your product launches,
you should see an improvement in user retention with younger cohorts as
the product improves. If this is not the case, you should consider
whether the hypotheses or features you are working on are the right
focus.

Most importantly though, this data will be the basis to give you a sense for your customer lifetime value
(CLTV). If you take the weighed retention data for the 6th or ideally
12th month and extrapolate it, you will get an approximation for the
average lifetime of your customers. Multiplying this with the average
revenue per account (ARPA) or respective plan that you are looking at
(e.g individual / team) it will give you your CLTV. This number is
really the quint essence of the cohort analysis, as it gives you an idea
about how profitable your business model is (=how much more money are
you making with than what you are paying to acquire him). Subsequently
it will also tell you the highest price you can spend on customer
acquisition to grow profitably. It is important to note here that
although super valuable, especially in the early stages of a startup
this number will always be an estimation and most likely not 100%
accurate. So keep in mind to continually track and fine-tune your CLTV
calculations.

And one last thing: If you have accounted only for
paid subscriptions as defined at the first question above, then the
base rates of each month will also give you the most accurate number for
paid customer growth and subsequently MRR growth. Two charts you will want to have at hand when talking to investors.

Q: Is that it?

For this post, yup! If you want to learn more about cohort analysis or SaaS Metrics, I would strongly suggest to check out Christoph’s and David Skok’s blog. And in case you have any questions on the above or something is unclear, feel free to ask away in the comments or send me a mail and I will do my best to answer you (or forward the hard questions to Christoph). ;)

4 comments:

bkrstovic
said...

Great post. Retention (cancellation) cohort analysis is an absolute must for any subscription-based business, especially for product and retention teams.

It would have been great if you covered a few scenarios that SaaS companies could directly benefit from. For example, we have retention cohort analysis for respective acquisition channels; so we can determine customer behavior patterns and quality of customer retention for each distinct group (inbound, promotion campaigns, PPC, and so on).