Using Product Analytics Percentiles In Forecasting, Trends and Anomalies
Percentiles are critical for making more informed product decisions, based on the actual usage of large groups of users. By comparing percentiles of different segments and behavioral cohorts, we can identify positive and negative patterns in our data over time. This can allow us to anticipate how future values might change based on current or past trends and make adjustments as necessary.
This is our second article from our series on using percentiles with product analytics. In this article, we’re going to dive deeper and show you how to forecast, discover trends and detect anomalies in percentiles based on product metrics.
As mentioned in the previous article, it’s very easy to get insight into percentiles and distributions of your metrics with Stormly. And more importantly, since it’s often not enough to look only at averages when it comes to product analytics, we’ve created the Percentiles and Distributions insight to make your analysis much more accurate.
You may ask why metrics such as “averages” can be misleading. We like to use product analytics metrics for a music streaming service as an explanation.
So, let’s say that it seems like most users only play around 3 songs on average — but if you glance past these statistics, you’ll realize that some individuals might very well spend hours listening without stopping while others almost never play anything!
This example shows how even though our measurements seem accurate on paper-in reality they aren’t always reflective enough to give us a true representation of what goes on within a product.
This is why Stormly’s Percentiles and Distributions Insight gives you insight into the full distribution of any metric, such as total songs played per user, conversion time per user, and much more.
So let’s see how you can calculate percentiles using “user” and “event calculation” reports, and even forecast or detect anomalies in percentiles.
User Calculations
Let’s start with calculating the 75th percentile of the total number of songs played per user. The 75th percentile in this case will mean that 75% of all users played equal or less than this amount of songs in total. Or conversely, 25% of users played more than this amount of songs.
We’re going to do this using the User Calculation from the Basic Reporting Insight. Click “Add Item” and choose “Event Count” and then the “Play Song” event. Then click on the “Total” dropdown option, and select “Percentile”. In the popup enter a custom percentile, 75 in this case.
Now we simply run our report and see the results of the 75th percentile of total songs played per user, over the last half a year.
As we can see, on most days 75% of users played a total of 2 or less songs, while only a few days saw a total of 3 or less songs played per user. While this is an interesting insight, we could have done this as well with the Percentiles and Distributions insight. This would give us even more metrics out of the box, such as plotting different percentiles, the min, max and averages together over time.
Both the “Percentiles and Distributions” and “User Calculation” insights work on the cohort date of users. By default this is the date the user was first seen using the app or website. But what if we want to get more details about the 75th percentile of a song’s playtime, plotted over time by the actual moment the song was played?
Event Calculations
To plot a numeric event attribute, such as playtime of a song over time, we need to use the “Event Calculations” insight from “Basic Reporting”.
Click the “Add Item” dropdown, and select “Event”. Then pick the “Play Song” event, and then the “Playtime” attribute. Change the dropdown on the left of the new item from “Total” to “Percentile per event” and enter 75 to get the 75th percentile.
Once the results are in, we can see that 75% of the playtime of songs played for most days of the last half a year is 250 seconds or less, except for one day in august where it is 300 or less.
Note that for event based reports, we can only calculate percentiles of numeric event properties. This is because for event counts, there is no variability in the number of events per event, as it will always be 1.
To get percentiles for the total event counts per user, or average of an event’s numeric property per user, we need to use the “Percentiles and Distributions” and “User Calculation” insights, as described in our first article.
Forecasting And Trends In Percentiles
What’s interesting about Stormly, is that anywhere we have a User Calculation or Event calculation option, such as in the forecasting and anomaly detection insights, we can use percentiles as our metric to work with.
In this case we’d like to forecast and see seasonality for the 95th percentile of the average playtime per song per user.
Open the Forecasting insight and choose “User Calculations”. Select from “Add Item” the “Events” menu and then “Play Song”, and finally select “Playtime”. From the dropdown menu on the left, change from “Total” to “Percentile per user” and enter 95 to get the 95th percentile.
By default, any numeric event property is totalled per user. In this case, we want the average playtime per user, so we have to click the gear icon, and choose the average option, as show below:
The last thing we want to do is to filter only by users that did at least play a song. We can do so by picking the “Play Song” event in the top bar user filter, and picking “Has this event”.
This makes sure that our percentile doesn’t take users with zero playtime into consideration. By default if a user doesn’t play a song at all, they will have a playtime of zero, and hence have lower averages or percentiles with much more zero’s.
Now let’s run our forecast. Once the results are in, to make things a bit nicer, we can choose the “Format” option from the top and choose “Relative Time” from the “Display As” menu option to make our playtime forecasts easier to read.
As we can see, for 95% of the users the average playtime per song hovers from 5 minutes up to 6 minutes and 40 seconds. Only a few days reach 8 minutes. For the next 7 days, it’s expected that this pattern hovers around 6 minutes.
Other things we can see, is the trend-line in the 95th percentile. We can see below that the general and forecasted trend is that the playtime will go down.
When it comes to day-of-week effects, we can see that on Thursdays and Saturdays, the 95th percentile is up to 2.5% higher than for other days of the week.
And that’s not all! You can also, for example, detect anomalies in the 95th percentile. This can be useful to automatically get notified when there’s a significant drop in the number of songs played per user, for 95% of users. It could indicate there’s some problem with the music streaming app, or other technical issues.
To set this up, an Event Calculation from the Anomaly Detection Insight is most suited, as it works by taking the 95th percentile of the day the play event was triggered.
Product analytics is essential to businesses of all shapes and sizes. By understanding how to use product analytics percentiles in forecasting, anomalies and trends, you can improve the accuracy of your analysis and make better decisions for your business.
If you’re looking for a tool that can help you with product analytics, Stormly is the best option. With Stormly, you can get started improving the accuracy of your forecasts right away. Sign up today and see what a difference it makes!