My game is down! All calls are returning NotAuthorizedByTitle error

Question

question

mfdemir3 asked Apr 3, '20

My game is down! All calls are returning NotAuthorizedByTitle error

Hi, my game started to reject all API calls suddenly. It started to happen around April 2, 2020 9:41 PM.

All calls are returning

{
    "errorCode": 1191,
    "code": 403,
    "errorMessage": "Action not authorized by title",
    "error": "NotAuthorizedByTitle",
    "status": "Forbidden"
}

I can't even call Admin/GetPolicy to check if policy is changed because it gives this error too. This error started to occur with no reason. I didn't do anything about the game or Playfab settings that day. I need urgent help please. We are losing all our players right now.

TitleID is D768.

apis

10 |1200

Attachments: Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Answer 1 · 2020-04-04T03:53:53Z

brendan answered Apr 4, '20

@mfdemir3

We've tracked down the issue and your title should be back to normal momentarily. Specifically, what happened was this: We have a "quarantine" vertical for titles that are calling endpoints at an excessive rate. In the free tier, titles are moved there if it's found their usage may cause a performance issue otherwise. If you have a look at your event log, you'll notice that you're generating a large volume of calls per player, as well as hitting a number of title_exceeded_limit errors each day.

That also happens to be the vertical that gets service updates ahead of the main vertical, after all testing has been completed. Unfortunately, there was an edge case that wasn't detected that blocked some API calls, based on the title's configured API policy. Your title happened to be one that exactly matched the pattern for blocking the calls you're using, which is what caused such an extreme outcome for you. Worse, our automated monitoring didn't catch this, as the overall volume wasn't high (while you have a large number of calls per player, it's a small total number of players). You have our sincere apologies for the impact - we're taking steps to make sure this issue cannot happen again.

Now, I should call out that I also noticed that you updated your studio to the new pricing model (presumably in an attempt to unblock the title). Since you're still under 10K total accounts in the game, you do have some time to bring down the total API usage, which I would strongly recommend. Reviewing the title's March using the new billing model (to be clear, no one is being billed for March using in the v2 model, regardless), it exceeded the Standard tier's included meters for both profile reads and Cloud Script executions by a small amount. Now, the title is still in development mode for now, so it'll a) not be billed, but b) block creation of player accounts once you hit 10K players. But I'm sure you can see that given that, if you were to scale up to a lot of players, the price would go up well past the base fee.

And if you did move to the v2 pricing specifically as a measure to try to fix this issue and would prefer to stay in the legacy Essentials tier while you work to optimize the title, let me know and I can get that changed back for you.

2

10 |1200

Attachments: Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

mfdemir3 commented · Apr 04, 2020 at 12:26 PM

Thank you Brendan. Yes I've moved to v2 pricing when I was trying to fix the issue. I would prefer to stay in the legacy Essentials tier while I work to optimize the title please.

0 ·

brendan mfdemir3 commented · Apr 06, 2020 at 08:45 AM

Thanks, I've asked the billing team to move you back. If you encounter any issues, please do let me know.

0 ·

Answer 2 · 2020-04-04T00:11:16Z

brandon@uprootstudios.com answered Apr 4, '20

@mfdemir3 In the Game Manager > Title Settings > API Features, is 'Disable Client API Access" unchecked?

10 |1200

Attachments: Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Answer 3 · 2020-04-04T02:05:09Z

Brent Batas (Lisk) answered Apr 4, '20

@Brandon Phillips I'm having same exact problem as OP. Disable Client API access is unchecked. I haven't pushed any online changes in about a week and it suddenly went all offline in the past hour.

3

10 |1200

Attachments: Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

brendan commented · Apr 04, 2020 at 02:22 AM

Brent, where are you seeing this? I'm looking at your title right now, and all the calls appear to be going through fine. In fact, you had a surge in Steam logins in the last hour or so.

0 ·

Brent Batas (Lisk) brendan commented · Apr 04, 2020 at 02:27 AM

It just started working again as you responded. We were seeing this on both client and lobby (servers). Pretty much every API call was returning Not Authorized. Nobody could play the game for I think close to an hour (

https://i.imgur.com/HGl5SXW.png and

https://i.imgur.com/EnqEKuu.png

)

0 ·

brendan Brent Batas (Lisk) commented · Apr 04, 2020 at 03:55 AM

We've updated the status page with more info on this. The issue is fixed, at this point. Sincere apologies for any issues caused by the disruption.

0 ·

question