PlayFab Retention Calculations (Is it rolling retention?)
Is Retention from the retention reports in Playfab calculated as rolling retention based off player installation time or are retention metrics based off a specific daily UTC reset/bucketing time.
Eg: Person installs 5 minutes before UTC time rolls over from Monday -> Tuesday, and that player played for more than 5 minutes. Are they counted as D1 retained? ...or is D1 retention calculated as a player logging events anytime between 24hrs after installation to 48hrs after installation.
checked out the docs and didn't wasn't sure by the wording: https://api.playfab.com/docs/tutorials/landing-analytics/analytics-metrics
more detail:
http://www.nielshoven.com/bad-bucketing-how-analytics-companies-are-getting-retention-wrong/
Best Answer
Well, the first thing I'd highlight is that for any non-trivial user base that's not limited to a small number of timezones, it's doubtful that the difference in retention between the two would be statistically significant. The theory is that the number of players that happen to log in (the measure of the start of a session, which is what's used for retention calculations) on both sides of the UTC 00:00 divide would be large enough, relative to the total user population, to significantly impact your retention numbers. That seems fairly unlikely, but we are working on a session auth update for later this year which will provide us with more flexibility around this sort of measurement. I'll file a backlog item with the data team to investigate that, once the new session system is online.
Actually, in thinking about this further, I would argue strongly against using a rolling, per-player window. As stated above, using a standard day (0 UTC, in our case) for all players means that you'll have a relatively small number of cases where play in one time window straddles two days.
But if you change that to a rolling window per-player, you're guaranteeing a high rate of these cases. By having the window for each 24 hour period starting exactly when the player first logs into your game, there's going to be a high correlation with their normal play times - which will include your per-player zero mark, but will extend before and after it by some indeterminate amount.
So in fact, using that model for retention would give you significantly worse data concerning anything beyond day one retention.
