Anatoly Volnov avatar image
Anatoly Volnov asked

Entity file data loss

About once every couple of weeks I get a report from a player that they've lost their data. In this game, save data is kept in Entity files (Title 82E3), and at a glance everything seems alright, but when trying to download a file from an affected account I get this XML:

<Message>The specified key does not exist.</Message>



At this URL:!4D4E73B99933EAD2/82E3/FB42F2209D8740AC/70FC5620B3CE7D20/garageData.c79e2123aa9e498dafc98916228cda25?X-Amz-Expires=600&x-amz-security-token=IQoJb3JpZ2luX2VjEJ%2F%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEaCXVzLXdlc3QtMiJHMEUCIFhSfBNBCuobEY%2BAj6u3nvrduqLCt1tCBWf0nf53xv9nAiEA20r7FDUxy%2FaGPc25MsJBl6WKszXgcNGrFO91CRAqhbgqvQMI%2BP%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FARAAGgwzODgxMjgwMzk2MzMiDBo4jSq08XLAPHLa9SqRA91HkIAe9p59ymR5tMufyA3UqjRe7TUqr467tcN6vdnZh%2FtWMvFlPH%2F0CDnMR8gWZkyRcDfTD2cUscCqLCbbAr8J2qdD1PT2R3hXM1zf1tYX3%2BRLZC6n%2FyRVM4W2iMJENa2YbK3eQx6fmib%2Ff1Zz2vaoL0aKq2BzYAtiQBxfGvhE4RE%2BD6XngtIXnxwUAcKaOY8sVk74dS7cdBdswbO2QMHgnaiFGtz6WLw58iIiZ%2B8dkm9eEjM9lnFpJ%2Fd8LHAi9JiSCHwv0AGouGL7OXsZvrESkuDV0KUf8JdJ%2Bf35cE3FKy33fzEH%2BzX3ikS908%2FyH5EHhLJI7MhHl8dYrFNlV8uE4xBQJd%2BzDjnfp96ZOP0UHlAZ1wwxU7nk3u6TAK4ADeqsE8k6POdUOCUubyj0qkaSUPgAFfdsWlRhXF5Dhg%2BBNoS7g0PRhYg14FglPWT8tQhrh2hDsUwvuQRJERekxYVkqe7AaXX0lf4zZEl44cEx3IKiUAOBRYJ%2Bd0izGzbbj3dBYi3gg5htG4PgTRfE4TnhMLupxvYFOusBftbi5VpXAT78ZqTJ0k1lmv1ROY1peMRlXimoazUvpIdRAFkhHNx6id97uKmgaKOgKz5r1EUAKv4zQKK9%2BWrdf5%2Bq4etN3lNJUHi7pDVuL3m9OqYpL1TtHRRl0Y%2ByWgKBpt8SErMmoILyCi0QOBMuGNooEJz7btAf3hMA2XolIVSRPeJEjDl4delfeOto66w%2FZSkYXvuIdWiBNqaZjADelUkA%2BVd1n%2FRgSZZUxIlklYMBjlZf9pjchceaMdBLf7ySMlk%2BWbdJWjqHnJS9aWE6UoP9iio5ZZuhvAHv7dQcQ0bciHmX4tfgGzvvag%3D%3D&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ASIAVUXR34LIQR4WI5J6/20200529/us-west-2/s3/aws4_request&X-Amz-Date=20200529T235932Z&X-Amz-SignedHeaders=host;x-amz-security-token&X-Amz-Signature=6782936413ce9216f64d7ec3b232c8ae7a1114319e6a242f44e9fc324ae3d0da

Following this error I have to either delete all of their files or make them a new account. I will make a new account in this case and orphan the old one, so that the affected one can be investigated, the Master ID is FB42F2209D8740AC.

Also I can't type after a code block when composing this message, can't get out of the block (on Safari).
10 |1200

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Anatoly Volnov avatar image
Anatoly Volnov answered

Hi @FranklinChen and thanks for getting back to me.

I haven't been able to isolate a pattern to this thing either, so I can't really provide a specific repro step.

Generally I've followed the tutorial on entity files and used the code from that for handling the uploads. Here's a text file with all the upload related code: uploadstuff.txt. I think some fingers could be pointed to the aborting parts since I have encountered files stuck as pending in Game Manager before, so that may be related to your #2, but that might also be due to the game crashing during an upload due to reasons unrelated to this issue. When I initially implemented uploading I wasn't very familiar with coding for a backend so there definitely may be some bad patterns there.

Generally uploading is called fairly often but it's always the same mechanism that's time-restricted as you can see in the code. All steps of the upload process are just daisy chained and called by the previous step as outlined in the tutorial. The first method that gets called is the paremeter-less UploadFiles().

Also it might be exacerbating the problem that the game crashes at all (which it does sometimes on some devices), and I might be better off waiting to upload until more crash-safe times.

uploadstuff.txt (6.7 KiB)
10 |1200

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

franklinchen avatar image franklinchen commented ·

I see, thanks for sharing your details. Let me forward your message to team, will talk to you later

0 Likes 0 ·
franklinchen avatar image franklinchen commented ·

Hi @Anatoly Volnov, sorry for the delay. I just received the feedback from our engineering team, they couldn't find a plausible root cause. There is a possible race condition - Files get automatically deleted if not finalized within 15 minutes. If FinalizeFileUploads is caused concurrently with this, upload can be finalized while the file is being deleted. However, this seems extremely unlikely, and based on your code, does not seem that there is a 15 minute wait between Initiate and Finalize.

For further troubleshooting, if you are still suffering from this issue, do they have more recent data? The impacted player id is helpful.

And once this occurs, is it possible to verify that file uploads completely stop working for the player or can you attempt a new upload?


0 Likes 0 ·
Anatoly Volnov avatar image Anatoly Volnov franklinchen commented ·

Hi, I haven't actually experienced this issue after releasing a game update that introduced some more waits and checks, as well as a slight tweak to finalization, including waiting for the upload to finish before doing anything 'tumultuous' in the game that could be crash-prone. I don't know what did it exactly, but I think the issue is gone. I haven't got a case like that in about a month I think.

I'll let you know if I get an affected user again, thanks.

0 Likes 0 ·
franklinchen avatar image franklinchen Anatoly Volnov commented ·

Thanks for your update, let me know if you got the same issue:)

0 Likes 0 ·
Citrus Yan avatar image
Citrus Yan answered

Hi @Anatoly Volnov,

We can confirm this issue in your title and have informed the engineering team about this, any updates from them will keep you informed.

10 |1200

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

franklinchen avatar image
franklinchen answered

Hi @Anatoly Volnov, we're investigating this, and see it happening to a number of your users. However so far we haven't been able to reproduce it or come up with a theory.

The user you provided hasn't had a successful file update since 4/20, though I see other players are in the correct state since then.

We do see one suspect thing in our source code, could you please confirm:

1. Is it possible to provide a repro step? If not, at least provide us with their call pattern for the Entity File APIs?

2. If there are conditions where player call abort, or start a pending operation but never complete it.

3. If the time between calling Initiate, performing the upload and then calling Finalize is less then a few minutes every time?

4. Or are there cases were player might call Finalize minutes later?

10 |1200

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Write an Answer

Hint: Notify or tag a user in this post by typing @username.

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.