question

Jimmie Tyrrell avatar image
Jimmie Tyrrell asked

Random spike in CloudScript execution times

My game is nearly ready to launch, but I am occasionally getting reports from beta testers of inexplicable network errors. I've finally managed to track them down, and noticed crazy high execution times in some ExecuteCloudScript calls. For example:

{
  "PlayFabEnvironment": {
    "Application": "logicserver",
    "Vertical": "master",
    "Commit": "3142600",
    "Cloud": "main"
  },
  "CloudScriptExecutionResult": {
    "FunctionName": "EndRound",
    "ExecutionTimeSeconds": 18.4062025,
    "ProcessorTimeSeconds": 0.0,
    "MemoryConsumedBytes": 292600,
    "HttpRequestsIssued": 0,
    "APIRequestsIssued": 0,
    "Revision": 192,
    "Error": {
      "Message": "The script execution was terminated after the maximum execution time limit",
      "Error": "CloudScriptExecutionTimeLimitExceeded"
    }
  },
  "FunctionName": "EndRound",
  "EventNamespace": "com.playfab",
  "EntityType": "player",
  "SourceType": "BackEnd",
  "Timestamp": "2020-11-03T04:42:07.2239080Z",
  "EventName": "player_executed_cloudscript",
  "EntityId": "AEE80E559241238C",
  "EventId": "a44934770d024b71a399971a37ab8a44",
  "TitleId": "6D4B7",
  "Source": "CloudScript"
}

and

{
  "PlayFabEnvironment": {
    "Application": "logicserver",
    "Vertical": "master",
    "Commit": "3142600",
    "Cloud": "main"
  },
  "CloudScriptExecutionResult": {
    "FunctionName": "StartRound",
    "ExecutionTimeSeconds": 65.5883784,
    "ProcessorTimeSeconds": 0.0,
    "MemoryConsumedBytes": 12096,
    "HttpRequestsIssued": 0,
    "APIRequestsIssued": 0,
    "Revision": 192,
    "Error": {
      "Message": "The script execution was terminated after the maximum execution time limit",
      "Error": "CloudScriptExecutionTimeLimitExceeded"
    }
  },
  "FunctionName": "StartRound",
  "EventNamespace": "com.playfab",
  "EntityType": "player",
  "SourceType": "BackEnd",
  "Timestamp": "2020-11-03T04:44:21.0086008Z",
  "EventName": "player_executed_cloudscript",
  "EntityId": "AEE80E559241238C",
  "EventId": "10735606d1414239903c2ab07f85b923",
  "TitleId": "6D4B7",
  "Source": "CloudScript"
}

18 seconds for one, and then a whopper 65 second execution time. The 65s StartRound request is particularly mind boggling, because it's an incredibly simple handler:

handlers.StartRound = function (args) {
  var currentRoundId = randomId();


  server.UpdateUserReadOnlyData({
    PlayFabId: currentPlayerId,
    Permission: "public",
    Data: {
      LastRoundId: currentRoundId,
      LastRoundStartTime: (new Date().toUTCString())
    }
  });


  return {
    RoundId: currentRoundId
  };
};

Taking a look at my Cloud Script Processing times, you can see these two failed requests just before 21:00. Around that time, I also noticed a DownstreamServiceUnavailable error, which I don't know is related. StartRound doesn't have any downstream services that I know of.

Anybody have any ideas what might be causing these high execution times?

1wgla.png (163.4 KiB)
6 comments
10 |1200

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Seth Du avatar image Seth Du ♦ commented ·

May I ask how often does this issue occur? I have tested your code, but didn't reproduce this issue. How do you define the "randomId()" function? Does it contain external HTTP request?

0 Likes 0 ·
Jimmie Tyrrell avatar image Jimmie Tyrrell Seth Du ♦ commented ·

Hi SethDu. randomId is a one liner with no loops or http calls. Just generates a random string

0 Likes 0 ·
Jimmie Tyrrell avatar image Jimmie Tyrrell Seth Du ♦ commented ·

Seems like every of my 30 beta testers have seen this problem at least once. But I’m not positive, I didn’t have the proper logging enabled to diagnose this issue before. Maybe a new report every day

0 Likes 0 ·
Evgeniy avatar image Evgeniy Seth Du ♦ commented ·
0 Likes 0 ·
Jimmie Tyrrell avatar image Jimmie Tyrrell Evgeniy commented ·

This is not great. I'm worried that adding users will actually exacerbate this issue. I wonder if Playfab is planning to address this?

@Evgeniy are you using Azure Functions for your app? I am not. Would be interesting to know if the downtime is on the Azure / CloudScript side, or the server API side.

0 Likes 0 ·
Show more comments
Seth Du avatar image
Seth Du answered

Thanks for the feedback. We have passed through previous Evgeniy's report to the team. According to the recent updates, this issue should be mitigated.

We will continue to monitor it and please feel free to let us know if you still have this issue.

10 |1200

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Jimmie Tyrrell avatar image
Jimmie Tyrrell answered

I've confirmed this is happening across multiple users. Probably the same problem @Evgeniy is seeing.

PlayFab team - I'm trying to determine my next move, and answering these questions would really help me out:

Is there a known issue causing these sporadic high execution times?

Is my game client misusing the PlayFab API and causing these spikes?

If I add more users to my game, will these spikes increase in frequency and duration?

If this is not my client's fault, will it be investigated and addressed?

If this issue will not be addressed on the server side, do you recommend I code an automatic retry into my game? I don't want to compound the issue by having all my clients retry at the same time, so I'll defer to your recommendation.

Thank you for your help!


gb4lxxl.png (416.0 KiB)
10 |1200

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Write an Answer

Hint: Notify or tag a user in this post by typing @username.

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.