When someone says “cohort analysis”, it is easy to imagine something complicated, something that is used only in special cases and requires fancy calculations.
In reality, you’d be surprised at how often this type of analysis is performed in mobile gaming – it is EVERYWHERE.
And it is used actively here at Hutch, trust me! Hutch is a lean development studio, so the testing, learning & iterating cycles are fast in here, and it pays off – this approach allows us to develop successful games and try out new things 😊 To derive actionable insights and make informed decisions in this environment, we heavily use cohort analysis.
In this article, I will tell you about different types of cohort analysis that can be encountered in the wilderness; then, I will cover the common difficulties & how to account for them.
Firstly, what is cohort analysis used for? I will not go over every possible case but mention the most common use cases
Cohort analysis for calculating LTV and other metrics:
Guess what retention, conversion, and LTV analysis are based on? Cohorts!
When you say that Day 7 retention on March 1st is 15%, it means that 15% of new users’ cohort from March 1st came back on Day 7. You put people into cohorts based on their install date and calculated their D7 retention – if that’s not cohort analysis, I don’t know what is.
The same logic applies to conversion on Day X, or ARPDAU (Average Revenue Per Daily Active User) on day 7 etc etc. Of course, in LTV’s case you aggregate your findings and produce a curve (a topic for a whole other article), but initially, you started with the cohort method by install date. Not too complicated after all!
Cohort analysis for user progression:
Whatever the game, it is always important to analyse and optimize user progression from level to level.
For that, you have to use cohort analysis. You need to put new users into cohorts based on their install time, calculate the % of users that progressed from level to level, analyse the time it took etc.
Mind how I said “install-time”, not “install date”. When analysing user progression, it usually makes more sense to take a longer time to cohort users by.
For instance, it is way more representative to analyse user progression based not on March 1st, but instead, look into all users who installed the game in March and see how they are progressing from level to level.
Cohorts for feature impact analysis:
If you make a change in early user experience, you want to know how it affects user behaviour.
For example, you ran a bunch of IAP offers that were targeting early users. Did this early exposure to good deals affect user behaviour in the future? Did it influence session length, future spending, improves short- and long-term LTV?
All these questions can be answered by taking a cohort of users that saw these offers in the early stages of their game, and a sample of users that did not have this experience (e.g., downloaded the game way before the offers were in the store).
In this example, comparing these two cohorts allows you to understand the impact of the new offer strategy.
You can test anything you want, but it is worth nothing if you don’t analyse the cohorts afterwards.
After the test is done, your job is to compare the behaviours of each test group, using confidence intervals (again a topic for another article).
This is 100% cohort analysis as well: one of the most reliable ones, as the users in the cohorts are randomized, so you don’t have to choose representative samples yourself.
Common cohort analysis problems and how to avoid them
Speaking of representative samples, we transition to the next part of this piece – common problems analysts face when using cohorts and how to avoid these problems.
Picking the wrong baseline group for cohorts comparison
If you are not AB testing, you need to make sure that cohorts you compare are comparable.
Let’s go back to our previous example: you are looking into the effect of a set of IAP offers targeted at new users.
You know the time during which these offers ran, you already have the cohort that saw the offers.
Now it is time to choose the baseline – a cohort of new users that did not see this set of IAP offers at the beginning of their experience.
You may be tempted to take a group of new users from the recent past and be done with it; however, some pitfalls can arise. You have to make sure the test cohort and the baseline cohort have users from similar sources and countries, and the distribution of these sources and countries are the same for both cohorts.
For instance, you cannot safely compare two groups of new users, one of which has 80% of organic new players, and the second one has only 10%.
So, remember – always try to clean the baseline as much as possible when performing a cohort analysis.
Not letting the analysis to mature to make conclusions
To properly analyse cohort behaviour, usually, you need to wait 😊
It depends on what you are after, but it’s important to keep it in mind. For example, you would like to know how getting higher rewards on day 0 affects future spending and engagement. Assuming you tested this on a cohort of users.
Let’s say that you are interested in short-term behaviour and would limit your analysis up to day 14 since install. To have enough data, you will need to wait for the cohort of interest to mature: namely, for a sufficient number of users to reach their day 14.
Of course, there is no shame in starting to look into the first days after install earlier, as valuable insights can be derived early on as well, especially if you notice a significant shift in user behaviour. However, it is always good to remember: check for maturity & be patient.
I hope I debunked some common myths about cohort analysis – now you can see how often it is used in every aspect of game analytics. It is nothing fancy or statistics-heavy; instead, it is one of the simplest and most robust ways of analysing human behaviour.
Anna Yukhtenko is Game Analyst at Hutch Games. A mobile video game publisher, creator of racing games including Rebel Racing, F1 Manager and Top Drives.
Before Hutch, Anna took analyst roles at Next Games, CPG company Danone and Bank of Finland. She owns a Master of Science in Economics from the University of Helsinki and a Bachelor degree in Economics from the Lomonosov Moscow State University.
For more insights from Anna, don’t miss this Mobile Mastermind article to learn how to monetize mobile games with subscriptions.