question

totaloncue avatar image
totaloncue asked

Regular Delays in Snowflake Data Ingestion

We are experiencing delays and/or data loss in access to our title(A0D3)'s raw event data starting Aug 1.

We run a query daily against the PlayFab shared database to pull all of the events related to our title and to store them in our own database for further querying. We see that for Aug 1 and Aug 2, there are < 500 events against our title stored in the shared database vs a regular event count of ~1 million.

The events do seem to be sent to the newly previewed Azure Data Explorer data warehouse. However, at the moment Snowflake is our primary access point for all our raw event data.

We are wondering if the issue is limited to our title or broader to the community; and also looking for guidance/help with resolving the problem.

Thanks

10 |1200
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

2 Answers

·
totaloncue avatar image
totaloncue answered

Thanks for following up @SethDu. We are on the Free Tier so do not have the ability to create tickets. Could someone check our title A0D3 to see if there is something causing a lag/delay in event ingestion to Snowflake?

We see that events up to Aug 3 have now caught up with the data we see in Azure Data Explorer but that the events for Aug 4 and beyond are still lagging significantly

For example, the total number of events fired on Aug4 and available for analysis on Azure Data Explorer are 1.6Million but we see only ~200 of those for Aug4 and beyond in the Playfab_Shared database for our title on Snowflake.

4 comments
10 |1200
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Seth Du avatar image Seth Du ♦ commented ·

We have forwarded the issue to our team, this thread will be updated if there is any feedback.

0 Likes 0 ·
totaloncue avatar image totaloncue Seth Du ♦ commented ·

Thank you. We appreciate any help with the issue.

In addition to the lag in data ingestion into Playfab_Shared, we are also seeing 1 order of magnitude increase in query execution time for pulling raw data from the shared database.The base query that we use to pull our title's data into our own table for further analysis has seen execution time go up from ~2 minutes to ~30 minutes with all other factors remaining the same.

0 Likes 0 ·
Seth Du avatar image Seth Du ♦ totaloncue commented ·

Your issue seems to be fixed. Please check it again and feel free to tell us if there is any other issue.

1 Like 1 ·
Show more comments
Seth Du avatar image
Seth Du answered

We don’t notice any issue on SnowFlake for now but I will inform our team to check it. If you are Pro or Enterprise Tier users and it is urgent, you may create a ticket via the guidance of this thread: https://community.playfab.com/articles/30527/playfab-support-overview.html.

10 |1200
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Write an Answer

Hint: Notify or tag a user in this post by typing @username.

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.