question

Trinidad Sibajas Bodoque avatar image
Trinidad Sibajas Bodoque asked

Validity of URLs from "InitiateFileUploads"

What's the validity of upload URLs obtained with "InitiateFileUploads"? According to the documentation (https://docs.microsoft.com/en-us/rest/api/playfab/data/file/initiate-file-uploads?view=playfab-rest), they are valid for 5 minutes, and in games we've been caching them for 4 minutes before obtaining a new one, to avoid doing the call every time and to upload the files as quickly as possible when apps suspend, and some months ago it worked with no issue.

However, at some point (not sure when) uploads have started failing ("SimplePutCall" returns success, but the uploads don't go through, and the call to "FinalizeFileUploads" indicates there's no pending operation to finalize). They seem to only succeed when the URL has not been used for an upload, regardless of how much time has passed. Has this changed at some point? What should we exactly expect regarding the validity of URLs?

Edit: code that reproduces the issue. Doing a file upload the first time, the console shows:

Starting the upload
Finalizing the upload
File upload success: meow

When doing the upload the second time, the console shows:

Starting the upload
Finalizing the upload
/File/FinalizeFileUploads: The file meow does not have an operation pending to finalize.
#if !DISABLE_PLAYFABENTITY_API && !DISABLE_PLAYFABCLIENT_API

using PlayFab;
using PlayFab.Internal;
using System;
using System.Collections.Generic;
using System.Text;
using UnityEngine;

public class EntityFileExample : MonoBehaviour
{
    public string entityId; // Id representing the logged in player
    public string entityType; // entityType representing the logged in player
    private readonly Dictionary<string, string> _entityFileJson = new Dictionary<string, string>();
    private readonly Dictionary<string, string> _tempUpdates = new Dictionary<string, string>();
    public string ActiveUploadFileName;
    public string NewFileName;
    public int GlobalFileLock = 0; // Kind of cheap and simple way to handle this kind of lock

    public string cachedURL = null;
    public string cachedURLFileName = null;

    
    
    
    void OnSharedFailure(PlayFabError error)
    {
        Debug.LogError(error.GenerateErrorReport());
        GlobalFileLock -= 1;
    }

    
    
    
    void OnGUI()
    {
        if (!PlayFabClientAPI.IsClientLoggedIn() && GUI.Button(new Rect(0, 0, 100, 30), "Login"))
            Login();
        if (PlayFabClientAPI.IsClientLoggedIn() && GUI.Button(new Rect(0, 0, 100, 30), "LogOut"))
            PlayFabClientAPI.ForgetAllCredentials();

        if (PlayFabClientAPI.IsClientLoggedIn() && GUI.Button(new Rect(100, 0, 100, 30), "(re)Load Files"))
            LoadAllFiles();

        if (PlayFabClientAPI.IsClientLoggedIn())
        {
            // Display existing files
            _tempUpdates.Clear();
            var index = 0;
            foreach (var each in _entityFileJson)
            {
                GUI.Label(new Rect(100 * index, 60, 100, 30), each.Key);
                var tempInput = _entityFileJson[each.Key];
                var tempOutput = GUI.TextField(new Rect(100 * index, 90, 100, 30), tempInput);
                if (tempInput != tempOutput)
                    _tempUpdates[each.Key] = tempOutput;
                if (GUI.Button(new Rect(100 * index, 120, 100, 30), "Save " + each.Key))
                    UploadFile(each.Key);
                index++;
            }
            // Apply any changes
            foreach (var each in _tempUpdates)
                _entityFileJson[each.Key] = each.Value;

            // Add a new file
            NewFileName = GUI.TextField(new Rect(100 * index, 60, 100, 30), NewFileName);
            if (GUI.Button(new Rect(100 * index, 90, 100, 60), "Create " + NewFileName))
                UploadFile(NewFileName);
        }
    }

    
    
    
    void Login()
    {
        var request = new PlayFab.ClientModels.LoginWithCustomIDRequest
        {
            CustomId = SystemInfo.deviceUniqueIdentifier,
            CreateAccount = true
        };
        PlayFabClientAPI.LoginWithCustomID(request, OnLogin, OnSharedFailure);
    }
    

    
    
    void OnLogin(PlayFab.ClientModels.LoginResult result)
    {
        entityId = result.EntityToken.Entity.Id;
        entityType = result.EntityToken.Entity.Type;
    }

    
    
    
    void LoadAllFiles()
    {
        if (GlobalFileLock != 0)
            throw new Exception("This example overly restricts file operations for safety. Careful consideration must be made when doing multiple file operations in parallel to avoid conflict.");

        GlobalFileLock += 1; // Start GetFiles
        var request = new PlayFab.DataModels.GetFilesRequest { Entity = new PlayFab.DataModels.EntityKey { Id = entityId, Type = entityType } };
        PlayFabDataAPI.GetFiles(request, OnGetFileMeta, OnSharedFailure);
    }
    
    
    
    
    void OnGetFileMeta(PlayFab.DataModels.GetFilesResponse result)
    {
        Debug.Log("Loading " + result.Metadata.Count + " files");

        _entityFileJson.Clear();
        foreach (var eachFilePair in result.Metadata)
        {
            _entityFileJson.Add(eachFilePair.Key, null);
            GetActualFile(eachFilePair.Value);
        }
        GlobalFileLock -= 1; // Finish GetFiles
    }
    
    
    
    
    void GetActualFile(PlayFab.DataModels.GetFileMetadata fileData)
    {
        GlobalFileLock += 1; // Start Each SimpleGetCall
        PlayFabHttp.SimpleGetCall(fileData.DownloadUrl,
            result => { _entityFileJson[fileData.FileName] = Encoding.UTF8.GetString(result); GlobalFileLock -= 1; }, // Finish Each SimpleGetCall
            error => { Debug.Log(error); }
        );
    }

    
    
    
    void UploadFile(string fileName)
    {
        if (GlobalFileLock != 0)
            throw new Exception("This example overly restricts file operations for safety. Careful consideration must be made when doing multiple file operations in parallel to avoid conflict.");

        ActiveUploadFileName = fileName;

        GlobalFileLock += 1; // Start InitiateFileUploads

        if (string.IsNullOrEmpty(cachedURL)  ||  fileName != cachedURLFileName)
        {
            cachedURL = null;
            cachedURLFileName = null;

            var request = new PlayFab.DataModels.InitiateFileUploadsRequest
            {
                Entity = new PlayFab.DataModels.EntityKey { Id = entityId, Type = entityType },
                FileNames = new List<string> { ActiveUploadFileName },
            };
            
            PlayFabDataAPI.InitiateFileUploads(request,
                (response) =>
                {
                    cachedURL = response.UploadDetails[0].UploadUrl;
                    cachedURLFileName = fileName;
                    OnInitFileUpload(cachedURL);
                },
                OnInitFailed);
        }
        else
        {
            OnInitFileUpload(cachedURL);
        }
    }
    
    
    
    
    void OnInitFailed(PlayFabError error)
    {
        if (error.Error == PlayFabErrorCode.EntityFileOperationPending)
        {
            Debug.LogError("InitiateFileUploads failed: PlayFabErrorCode.EntityFileOperationPending");

            // This is an error you should handle when calling InitiateFileUploads, but your resolution path may vary
            GlobalFileLock += 1; // Start AbortFileUploads
            var request = new PlayFab.DataModels.AbortFileUploadsRequest
            {
                Entity = new PlayFab.DataModels.EntityKey { Id = entityId, Type = entityType },
                FileNames = new List<string> { ActiveUploadFileName },
            };
            PlayFabDataAPI.AbortFileUploads(request, (result) => { GlobalFileLock -= 1; UploadFile(ActiveUploadFileName); }, OnSharedFailure); GlobalFileLock -= 1; // Finish AbortFileUploads
            GlobalFileLock -= 1; // Failed InitiateFileUploads
        }
        else
            OnSharedFailure(error);
    }
    
    
    
    
    void OnInitFileUpload(string uploadURL)
    {
        Debug.Log("Starting the upload");

        var now = DateTime.UtcNow;
        string payloadStr = now.ToLongDateString() + ", " + now.ToLongTimeString();
        var payload = Encoding.UTF8.GetBytes(payloadStr);

        GlobalFileLock += 1; // Start SimplePutCall
        PlayFabHttp.SimplePutCall(uploadURL,
            payload,
            FinalizeUpload,
            error => { Debug.Log(error); }
        );
        GlobalFileLock -= 1; // Finish InitiateFileUploads
    }
    
    
    
    
    void FinalizeUpload(byte[] _)
    {
        Debug.Log("Finalizing the upload");

        GlobalFileLock += 1; // Start FinalizeFileUploads
        var request = new PlayFab.DataModels.FinalizeFileUploadsRequest
        {
            Entity = new PlayFab.DataModels.EntityKey { Id = entityId, Type = entityType },
            FileNames = new List<string> { ActiveUploadFileName },
        };
        PlayFabDataAPI.FinalizeFileUploads(request, OnUploadSuccess, OnSharedFailure);
        GlobalFileLock -= 1; // Finish SimplePutCall
    }
    
    
    
    
    void OnUploadSuccess(PlayFab.DataModels.FinalizeFileUploadsResponse result)
    {
        Debug.Log("File upload success: " + ActiveUploadFileName);
        GlobalFileLock -= 1; // Finish FinalizeFileUploads
    }
}
#endif
11 comments
10 |1200

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Rick Chen avatar image Rick Chen ♦ commented ·

Did you followed this document Entity files and the SimplePutCall function in Unity? I will do a test on Unity and will let you know the result. For now, you could use Postman to call the API and upload the entity files. According to my test, the file uploading works on Postman.

0 Likes 0 ·
Trinidad Sibajas Bodoque avatar image Trinidad Sibajas Bodoque Rick Chen ♦ commented ·

Yes, I had tried that example and it works, because it gets a new URL every time ("InitiateFileUploads"). What I'm asking is to clarify when that URL is invalidated and on what behaviour we can rely now and in the future, because according to the documentation it's valid for 5 minutes, and caching it (to make a quicker upload when the app is suspended) worked some time ago for multiple uploads, but now it seems to only work for a single upload, so something seems to have changed in that regard.

0 Likes 0 ·
Rick Chen avatar image Rick Chen ♦ Trinidad Sibajas Bodoque commented ·

Yes, the storage service has been changed from AWS to Azure. And the upload URL should be valid in the 5 minutes as the document states. You could obtain a new upload url using the InitiateFileUploads API if the previous one has expired. You can find the expiration time of the upload url in the query parameter “se” of that URL.

If you have specific requirements, you may upload a zipped file, otherwise you need to upload one by one.

0 Likes 0 ·
Show more comments
Show more comments
Trinidad Sibajas Bodoque avatar image Trinidad Sibajas Bodoque Rick Chen ♦ commented ·

(It seems I can't directly reply to your last comment, so I reply to the first one)

Let's say I want to upload the file "meow" twice, and I follow these steps:

1. I call "PlayFabDataAPI.InitiateFileUploads" to get the URL. It succeeds and "resultCallback" is executed.
2. From the callback I cache the URL in a variable to use later. And I call "PlayFabHttp.SimplePutCall" with that URL. It succeeds and "successCallback" is executed.
4. From the callback I call "PlayFabDataAPI.FinalizeFileUploads". It succeeds and "resultCallback" is executed, so all good.


(Continued in next comment)

0 Likes 0 ·
Trinidad Sibajas Bodoque avatar image Trinidad Sibajas Bodoque Rick Chen ♦ commented ·

(Continuation from previous comment)


5. I wait a bit of time after the previous steps succeeded (less than 5 minutes, just a few seconds or a pair of minutes, doesn't matter).
6. I call "PlayFabHttp.SimplePutCall" with the URL cached previously. It apparently succeeds and "successCallback" is executed.
7. From the callback I call "PlayFabDataAPI.FinalizeFileUploads". It fails and "errorCallback" is executed, with the error code "NoEntityFileOperationPending", error message "The file meow does not have an operation pending to finalize.", HTTP code "400", HTTP status "BadRequest". The file in PlayFab's dashboard is the first one, not the one in the second attempt.

Doing this some time ago worked, the second upload with the cached URL was properly done and "PlayFabDataAPI.FinalizeFileUploads" succeeded.

I have added to the question some code that reproduces the issue, it's basically the code in https://docs.microsoft.com/en-gb/gaming/playfab/features/data/entities/entity-files, but with the caching of the URL added.

0 Likes 0 ·
Rick Chen avatar image Rick Chen ♦ Trinidad Sibajas Bodoque commented ·

I see. When you use the FinalizeFileUploads API, the backend will verify that the files have been successfully uploaded and moves the file pointers from pending to live. And the upload url should be closed. This is by design. You should use the InitiateFileUploads and get a new upload url to upload new file. But before calling the FinalizeFileUploads API, you can upload files to the same url multiple times, as long as the URL does not expire, and when you call the FinalizeFileUploads API, it will verify the last file uploaded.

0 Likes 0 ·
Show more comments

0 Answers

·

Write an Answer

Hint: Notify or tag a user in this post by typing @username.

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.