Despite all the effort in improving software quality, sometimes the unexpected happens, and as a game analyst, you’ll have to deal with a crisis situation.
When you work as a games/product/business analyst, it’s very likely that you start your day with a cup of your favorite caffeinated beverage and a monitoring session. You go through your main KPIs, your favorite dashboards, and then, on a crisis day, you’ll notice that something is off. Ideally, some alert systems have already tipped you off that there’s smoke in the room, or perhaps someone else noticed the situation first.
Depending on the seriousness of the crisis, (namely the potential impact on the bottom line) your heart flutters, and your brain releases dopamine rewarding you for detecting that something is amiss, you feel a sudden rush of adrenaline through your body in a moment of “fight or flight” response. You’ll need all these chemicals to see this day through. Time to put on the detective cap.
Start with Why
Besides being the title of an amazing book, by Simon Sinek, “Start with Why” is your first mission after you notice that something is happening with your product/game.
Your mission is to help your team understand the reason(s) behind some KPIs being too low. If you’re able to pinpoint the cause, the team will able to fix the issue more quickly. Or at least, there will be a strategy to deal with the consequences.
Therefore, your first priority is to formulate hypotheses for whatever might be causing a negative impact on your product. In order to formulate better hypotheses, you should try to understand:
- Are there any bugs reported that could affect the KPIs?
- Who is affected/How many players are affected?
- Is the negative impact correlated with an update/hotfix?
- Is it possible that something was introduced in a previous update, a previous AB test that started a snowball effect?
- Are there any unforeseen seasonality/black swan events that could explain this crisis?
Customer Support / Community are your right arm
I’ve learned that empathy and exposure to players/users are essential to get to the core of the player experience. And there aren’t any people in the teams with better exposure to the players/customers than customer support (CS) and community managers.
Asking CS whether there was an increase with a certain type of ticket, whether there were any noteworthy bugs reported can speed up the time it takes to understand whether there’s a bug. You’ll be able to use this source of information to start formulating hypotheses or to add support to the current hypotheses that you’re considering.
Communication and visibility
Most companies have certain procedures to deal with incoming product crises. Regardless of the procedure, you should be a continuous source of information, for both the team and stakeholders.
You should get the team involved in your hypotheses and the overall investigation process. Let them be aware of any new information that could help them solve/fix the situation.
You should also keep all stakeholders throughout the current investigation. Keeping everyone up-to-date makes it easier to report the long-term impact once the issue is fixed.
Divide and conquer
Segment, segment, segment, segment… All potential ways that you can think of segmenting users taking into account your running hypothesis should be considered. Here are some typical examples, in case you’re panicking and your creativity is failing you in this moment of need, here’s a list you can refer back to (Pro tip: create your own “flight check” segmentations, that you can run through to make sure you cover all bases):
- By Device. Are users on different platforms being affected the same way? This means checking whether the main KPIs that seen a negative impact behave differently whether a user is playing on desktop vs. mobile, iOS vs. Android, iPhone 9 vs. iPhone 12.
- WiFi vs. 4G. Are there players on WiFi being affected differently than players on 4G?
- By Country/Region. Self-explanatory, but it can be helpful when working on hypotheses that involve seasonality, game server location, etc…
- By user acquisition source. If the issue is only affecting new users, it’s crucial to understand whether we have any differences in KPI behavior between Organic vs. Paid, Affiliate vs. Marketing campaign… Have the volumes changed significantly over the past days (consider a period of at least 30 days to detect these changes)
- AB tests. If you’re running multivariate tests/experiments, don’t forget to check whether a specific variant is causing the crisis at hand.
Correlate and eliminate
Segmentation is useful to understand and find evidence regarding which players have been affected, thus helping narrow down the search. But we shouldn’t forget good old technical chart analysis to understand what could have caused the issue at hand. Here we use the same principles as Single-Subject design experiments and using visual analysis to quickly understand whether there are correlations with events that could affect the state of the game (Systematic Use of Visual Analysis for Assessing Outcomes in Single Case Design Studies).
In order to achieve this, we want to map relevant KPIs over time and correlate baseline movements with certain events.
Pro tip: have all members of the team contribute to map/record specific events that can potentially affect the baseline. This is incredibly helpful when you have to look back and correlate baseline movements with specific events. In case you need inspiration, here’s a list of common things to look out for:
- Marketing initiatives and featuring. We want to check when some marketing initiatives happened, or whether there was a viral/featuring event. These events often lead to a big influx of new players who might not be the best fit for the product. It’s normal that during and after these events some early KPIs will suffer.
- AB tests. Besides segmenting current ab tests, it’s often relevant to go back in time and understand whether rolling a variant could have had unpredictable negative long-term effects.
- Game updates, hotfixes. More often than not game updates or patches introduce bugs that will be the root cause of a crisis. Be especially inquisitive when it comes to unforeseen long-term effects of previous updates. When you feel that your product knowledge is not the best, it’s important to go through all changes introduced and ask team members whether they think that something is negatively impacting the game.
- Known bugs. Sometimes there are well-known “pet” bugs. These were once labeled low priority and relegated to the oblivion zone of the bug backlog. Some of these bugs come back to haunt the teams as they find new synergies with new content, or the landscape changes, and their relevance increases.
- Liveops. Live modifications or in-game events.
Depending on the nature of the crisis, your game analyst job and priorities might change for a period of time from a couple of days to weeks. Therefore, I added two more articles to this series that focus on the preventative steps, as well as how to deal with the aftermath.
There’s the 5P mantra: Proper preparation prevents poor performance. Learn more about how to prepare for a crisis in the next part of this series on Game analyst tools to prevent and solve a crisis.
If you’re ready to deal with what comes once a product crisis has been averted, go to the third and final part of this series: Game analyst guide to deal with the aftermath of a crisis.
Are you an aspiring game analyst and this post left you craving for more? Check my post on Breaking into the Games Industry as a Game Analyst.