question

Benjamin Golinvaux avatar image
Benjamin Golinvaux asked

Best architecture for long-term event analysis

Hello

I am a bit lost when it comes to analytics and report generation.I have added a set of custom events related to player behavior (the kind of character that has been selected, duration of a match, etc.) The events are correctly stored and I can perform Kusto queries to count them, filter them...


Now, I need to provide the game designers with dashboards that allow them to explore statistics built on these events.

However, my understanding is that the events are only kept for 30 days. I hesitate to invest time into building advanced Kusto queries, because it will be impossible to execute queries for long-time (> 1 month) trends in the game.

I have added an automated S3 backup that saves all events to a bucket, and I have checked that the JSON files can correctly be retrieved from these archives, but working only from this data will not scale really well when the number of players will (hopefully) increase.


A possible option would be to automatically run queries that perform some sort of data compression by turning the raw events into statistics, as a first step, then pushing these "summary events" into a database that I would manage on my own.


My question: in case the "compute summary of events" described above is a good solution, are there available mechanisms to automatically execute kusto queries that perform counts, averages... and export them to some endpoint, with production-grade quality? (automatic retry in case of failure, monitoring, alerts...)?Is this something that I should setup an external server / cloudscript functions to trigger?


More generally, if you have some interesting resources to share about the typical architecture of an analytics reporting system with lots of users, I'd be very grateful.


I realize my question is broad, but any advice, even anecdotal, is welcome.

Thank you very much in advance!

--Benjamin

analytics
10 |1200

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

1 Answer

·
Made Wang avatar image
Made Wang answered

>>However, my understanding is that the events are only kept for 30 days.

The number of days for data retention can be modified, please refer to Performance and retention - PlayFab | Microsoft Docs for its billing model.

>>are there available mechanisms to automatically execute kusto queries that perform counts, averages... and export them to some endpoint, with production-grade quality?

Do you want to execute some query statement when exporting event data? If so, PlayFab supports using custom queries to export specific data, you can check Use custom query in Event Export and edit the query statement . Or you can try to implement Kusto SDK in Azure Function to execute query and export data to specific endpoint, and you can use scheduled task to trigger Azure Function.

10 |1200

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Write an Answer

Hint: Notify or tag a user in this post by typing @username.

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.