Like true nerds 🤘
A quick note about me: I am Luca and I work as a Data Scientist at Bending Spoons (yes, we are hiring in ASO/Marketing!) where I also coordinate the App Store Optimization team.
This story starts like many other stories, with me on my desk, in the morning, before the first cup of coffee of the day, and with my brain still in “autopilot mode”. One of our project managers comes at me, freaking out because an app suddenly lost 20–30% of its downloads overnight, without any particular reason (middle of the week, no ASO updates, no nothing — at least, apparently).
After quickly calming my colleague, with a fake sense of tranquillity that was mainly given by the fact that I was still caffeine sober, I decided to follow my usual ASO problem-solving scheme to understand what could have happened:
- Check if it is a reporting issue (timeframes, partial data, wrong integrations). Nope, everything was fine.
- Exclude that it is a change in paid UA traffic. Nope, we didn’t have ads on that app. Only organic downloads, mostly coming from searches in the App Store.
- Understand which country has the sharpest decrease in downloads. Is it maybe bots in some remote, low ARPD country favoring our competitors? It didn’t look like, and US was the country suffering the most.
- Check if there are signs of seasonality trends in the search ads popularity of the keywords for which we were ranking in high positions. Nope, nada, nichts, all as flat as flat can be. Heck, I even checked at the bank holidays of the various countries where our downloads were decreasing. Nothing even there.
- If everything else fails, it’s finally time to get a coffee (😙) and go back to the basis by analyzing rankings. The total downloads you make after all are the sum all the downloads make in the various searches, so in theory if we are tracking all the keywords for which our app ranks we should be able to quickly identify the issue.
Luckily, at BendingSpoons, we have developed Crystal and Pico, two amazing internal platforms designed by our wonderful data engineers that track a huge quantity of App Store data, including search rankings. And you might think that this is how the story ends: I find a few searches that caused the download drop and quickly send an ASO update to apple to promptly correct the problem. “Ah you fool! Not so easily.”
Sometimes the more data you have, the more it is difficult to make sense out of it. Out of the thousands of searches for which our app was ranking on that day hundreds changed their value, some improving, other worsening. Moved by my big curiosity I then opened a Jupiter notebook, pulled some ranking data from our Big Query database and plotted it. This is a graph of just a few, handpicked rankings I tried to analyze.
Our very understandable app rankings. Ah! This is really so very very helpful.
Headache. It was the right time for my second coffee.
While going back from the coffee machine to my desk, I opened a statistics/machine learning book that always gives me inspiration in these cases. I like to call it “The Bible” because it is a 750 pages manual that covers mostly every method of statistic or machine learning ever devised by the great statistical minds that populated our world — and, even though I probably fully understand just below 10% of it, it always gives me great hints.
This time I decided to use a very famous technique called Principal Component Analysis, to see if there was any way to get a better representation of the above graph. Basically what PCA does is to try to find linear combinations of the starting variables to generate a much smaller set of factors that are uncorrelated between each other. In this case what we hope is that the app search rankings are highly correlated one with the other, and that there is a way of summarizing them effectively with just a “few lines”.
Total % of variance explained by factor. High variance explained by few factors = good.
And luckily for us, I found that it’s enough to have 3 factors to account for 90% of the total variance of the original variables. In other words, 3 lines can explain most of what is going on in the messy graph of the ranks above.
Factor time series with switch-points
Every line represents a group of searches (provided that you find the correct rotation for your PCA loadings 🤓) and as you can see there are some interesting patterns, as searches “loading” to the same factors moved together in interesting ways.
Solved a problem, but found a bigger one.
I was finally able to make sense of this. Thanks to this principle component analysis, by analyzing the correlation between the original rankings and the factors I plotted above, I managed to detect that the issue was in a particular word that was giving us a lot of downloads but that was only present in our keywords field. I then quickly sent an update to correct the problem, adding this word also in the subtitle. This luckily solved our issue 🙃.
Nevertheless, I was still very curious as it seemed like our app had many moments of unexpected changes in rankings, some in which we lost positions for some keywords, some for which we gained. While we might not have noticed them before because the effect on downloads were negligible (e.g. rankings that changed were too high to bring our app downloads anyways) some others might have had.. Do you have anything for us ThomasBCN? Oh yes of course you do! 😉
Our favorite ASO expert tweeting. Yes, Thomas BCN ❤️ ASO.
It looked like I came across something well known by the ASO community, the so-called “Keyword Dances”.
Keyword dances timing
Another interesting discovery is that “Keyword dances” seem to be too well distributed in time to be simple coincidences.. They seem to be of different types and they seem to happen at “almost-fixed” (and therefore possibly predictable?) intervals in time:
Random or.. not?
And to confirm it we can see this, with more or less precision (our rankings might not precisely attributed to the day) on a totally different app:
Factor analysis for a completely different app
I will continue working on these things to try figure out what actually happens during the keyword dances, but in the meanwhile I hope to have shed some light on the matter, allowing you, ASO friends, to give explanations a bit quicker to yourselves/your bosses/your customers.
The end
Whoa, you made it to the end 🙌. Cool, one last thing, remember to clap if you found this article useful! 👏👏
