Intermittent 504 errors from CloudScript - "third-party downstream provider returned an error"

Question

question

Cameron Duff asked Jun 18, '21

Intermittent 504 errors from CloudScript - "third-party downstream provider returned an error"

Hi there. I've had a project in development using PlayFab for a couple of years, but only recently have I been getting these GatewayTimeout 504s from various CloudScript calls.

/Client/ExecuteCloudScript: A third-party downstream provider returned an error to the PlayFab service

I don't interface with any other parties from my CloudScript other than PlayFab itself, and most of my handlers just call server.GetUserReadOnlyData or server.UpdateUserReadOnlyData once or twice each at tops -- I'm not hammering any of the APIs.

For example, a moment ago I had the error from a single call to a CloudScript handler with one instance of each of the above calls. Now that I'm play-testing and close to release, I'm noticing a few of these errors every day from different CloudScript handlers.

Is this an issue with the performance/reliability of PlayFab's servers at the moment, or could it be something else?

Thanks in advance!

CloudScript

8

10 |1200

Attachments: Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Rick Chen ♦ commented · Jun 18, 2021 at 05:47 AM

Could you please provide your title id for us to diagnose? Do you know which CloudScript Function is causing the error? Please also check this thread and see if it helps: https://community.playfab.com/questions/25614/occasional-error-on-api-calls-a-third-party-downst.html

0 ·

Cameron Duff commented · Jun 18, 2021 at 06:13 AM

Hi @Rick Chen, thanks for responding. My title ID is "(113C0) Dragonfist Limitless".

I've experienced this 504 from multiple different handlers in my CloudScript, 2 of which I know for certain ottomh: AddPlayerExperience and SynchroniseInventory.

I don't know which specific server call within my handler is causing the 504 as it's just a HTTP response error rather than a stacktrace, afaict. I saw that thread you linked earlier and also looked at a few others, but none seemed applicable--for example, my AddPlayerExperience handler is specifically rate limited to 1 per 5 seconds, so I'm not hammering it and don't think it should be locking anything up.

My AddPlayerExperience CloudScript handler calls:

server.GetUserReadOnlyData (once every time)

server.UpdateUserReadOnlyData (once every time)

server.GetLeaderboardAroundUser (once every time)

server.UpdatePlayerStatistics (once sometimes)

server.WritePlayerEvent (once sometimes)

It never hits WritePlayerEvent and UpdatePlayerStatistics in the same instance, so at max there's 4 server calls, as far as I can tell, although I imagine the issue has more to do with one of them individually than the number of them in sequence.

Cheers

0 ·

Rick Chen ♦ Cameron Duff commented · Jun 22, 2021 at 02:07 AM

I have checked your title from our backend, despite there was a spike of the DownstreamServiceUnavailable error at around 2021-06-17 10:00 (UTC), there was no incident or custom report about this error. It is hard to find out what caused that spike of error. The CloudScript in your title is complicated and it is hard to reproduce the error. Is this issue still happening? If so, is it happened occasionally or every time? Could you provide a reproducing steps of this issue?

0 ·

Cameron Duff Rick Chen ♦ commented · Jun 22, 2021 at 02:41 AM

Hi Rick. Bummer. I've got 3 records of receiving the 504 GatewayTimeout error at different points. I think I've seen it more than this but it may have been earlier than my event history allows viewing.

These are all UTC times:
05/25/2021 5:08 AM - AddPlayerExperience handler
06/07/2021 4:40 AM - AddPlayerExperience handler
06/17/2021 2:19 AM - SynchroniseInventory handler

Might have been a coincidence that it happened on the days I happened to be testing. My concern is just that it might be something more regular once I have players actively playing.

Apologies about the CloudScript -- it's all transpiled from TypeScript & minified. I could provide the relevant handlers to you unminified (in either the transpiled JS or TypeScript, as you prefer) via email if it would help, but perhaps for now it will be better just to see if you're able to see anything in your server logs for the timestamps above, and I'll keep you posted if it is an ongoing issue rather than spending a lot of time investigating at this stage.

Thanks again for your help!

0 ·

Show more comments

Rick Chen ♦ Cameron Duff commented · Jun 25, 2021 at 03:48 AM

Thank you for understanding. I am looking forward to any follow-up.

0 ·

question