question

Paolo Sy avatar image
Paolo Sy asked

Matchmaking tickets not being received by queue after a few hours of operation,Matchmaking tickets not being received by queue

Hello,

Our game, with matchmaking implemented, has been in open beta for more than a week now.

It was working fine when:

1. We had to make a new queue for an update.

2. We made the update (client) point to that new queue.

3. It worked for a day or so.

After that, for a period of time, it stopped receiving tickets (we can see this via Matchmaking (Preview) > our_queue_name > Usage) and Tickets Waiting for a Match, both regular and backfill, is at 0 even though there are tickets that started matchmaking for that queue (e.g. players are queueing and reported that the queue timed out and they never got a match). We even tried to check it by queueing in the game ourselves and the statistic in the usage is 0 even after a few minutes of refreshing it. Usually we see the stat update after 5 mins of refreshing the Usage graphs.

The problem was temporarily solved when we tried creating a new queue for it, rename it so that it would be the queue used by the update/client. We even made our plan pay-as-your-go to be sure and even though we did not go beyond our Limits (based on the information the Limits page showed us). We are able to go live again, see players queue and get matched. It would work for a while then maybe after two hours or so, players will queue and will not be matched and we see that Tickets Waiting for a Match are 0 again.

This felt like an emergency situation and it felt very helpless not being able to know the cause. Did anyone have a similar situation?

Any help would be appreciated. Thank you.

limitsMatchmaking
5 comments
10 |1200

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Paolo Sy avatar image Paolo Sy commented ·

Hello I apologize, I think I must have pasted the question twice. If I can edit the post and delete the latter half that would be great. Thank you!

0 Likes 0 ·
Gosen Gao avatar image Gosen Gao Paolo Sy commented ·

May I know your Title ID?

0 Likes 0 ·
Paolo Sy avatar image Paolo Sy Gosen Gao commented ·

Hello, 78e22

0 Likes 0 ·
Show more comments
Gosen Gao avatar image
Gosen Gao answered

>> For every server with incomplete number of players during backfill, we call a CloudScript every 10 seconds until the game is full (getServerBackfillTicket). Question is, does this count to the API limit you are saying?

GetServerBackfillTicket has its own request limit(10 calls per min). You can check it on [Game Manager] -> [Title Settings] -> [Limits]. Please note, this API can be called from your server, you don’t have to execute a CloudScript to call it. It will increase the cost.

>> Also, regarding 100 calls per 2 minutes limit, is this applicable only to the Free Plan?

The 100 calls per 2 minutes limit is a client side API request limit, which is same to all titles in public clusters.

>> Has this change from free plan to pay-as-you-go plan not been always instant? Is there any way to accelerate this process? Does having a pay-as-you-go plan solve these possible limit problems? We'll definitely make some changes to the frequency of the calls for cost efficiency but now that the game is live, will making our account pay-as-you-go help temporarily solve the throttling issues?

Plan’s change will take effect next month. Besides the Enterprise Plan, there's not much difference in limits between plans. The most obvious difference of launching a title from development mode is the limit on the number of users. You can check the difference in this page: https://playfab.com/pricing/.

Changing to pay-as-you-go plan cannot solve this issue. If you have measured the cost, and it is possible for you to upgrade to Standard plan. I would suggest you to upgrade to the Standard plan so that you can submit a support ticket to us. With SLA agreement, we can provide fast-response, dedicated and private technical support for you.

To help us dig into this issue, could you please provide the detailed info about the queue’s settings? And we also want to know how your game’s Matchmaking system works?

1 comment
10 |1200

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Paolo Sy avatar image Paolo Sy commented ·

Okay, let me get back to this as soon as I can to detail the queue settings and our system.

0 Likes 0 ·
Gosen Gao avatar image
Gosen Gao answered

After looking into your title, we found that there are plenty of CloudScriptExecutionTimeLimitExceed errors occurring in your title. Are you using CloudScript to create Matchmaking Ticket? Too frequent requests on a CloudScript function can cause throttling and make it exceed the API limit(100 calls per 2 min), then caused the creation of the ticket to fail. May I know the detailed error callback from PlayFab so that we can dig into it?

We also notice that a few errors occur when your game try to call API CancelMatchmakingTicket but the MatchmakingTicket is already completed. We are unclear about this error because the client should poll tickets from time to time to get the latest state of tickets, this error can be avoided.

There are also some errors from DeregisterGame API, which is designed for Legacy Multiplayer Server. Are you using Legacy Server in your game?

2 comments
10 |1200

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Paolo Sy avatar image Paolo Sy commented ·

Hello!

Thank you so much for looking into our title.

We're not using CloudScript to create a matchmaking ticket. Tickets are created via client calls (PlayFab for UE4).


The CloudScriptExecutionTimeLimitExceed errors come from calling a CloudScript that calls a request to an external API:

  • When a match is found, runs CloudScript: Automation > CloudScript > Revisions (Legacy) > matchFoundV2, which calls an external HTTP request (the logic of this request is that it spawns a server and waits for it to finish spawning).
  • We do not need the response; we just need the CloudScript to fire the request.
  • The said CloudScript is only executed only when a match is found, which in our current game traffic, only happens around less than 10 times (that means < 10 API calls) in 2 mins. We don't think this is the cause due to the small number of calls it takes.
0 Likes 0 ·
Paolo Sy avatar image Paolo Sy commented ·

After careful investigation, we found that we have a lot of player_executed_cloudscript calls in the dedicated server (especially when backfilling match is in process).

  • For every server with incomplete number of players during backfill, we call a CloudScript every 10 seconds until the game is full (getServerBackfillTicket). Question is, does this count to the API limit you are saying?

Also, regarding 100 calls per 2 minutes limit, is this applicable only to the Free Plan? If so, we recently (a day before this question is posted) put our studio plan to pay-as-you-go, but in studio settings, the pay-as-you-go plan is not reflected immediately (and still is right now):

  • Current Plan: Free plan
  • Effective December 1, 2021: Pay as you go plan

Has this change from free plan to pay-as-you-go plan not been always instant?

Is there any way to accelerate this process? Does having a pay-as-you-go plan solve these possible limit problems? We'll definitely make some changes to the frequency of the calls for cost efficiency but now that the game is live, will making our account pay-as-you-go help temporarily solve the throttling issues?

Thank you for taking time to help.

0 Likes 0 ·
Paolo Sy avatar image
Paolo Sy answered

Hello!

Thank you so much for looking into our title.

We're not using CloudScript to create a matchmaking ticket. Tickets are created via client calls (PlayFab for UE4).

The CloudScriptExecutionTimeLimitExceed errors come from calling a CloudScript that calls a request to an external API:

  • When a match is found, runs CloudScript: Automation > CloudScript > Revisions (Legacy) > matchFoundV2, which calls an external HTTP request (the logic of this request is that it spawns a server and waits for it to finish spawning). We do not need the response; we just need the CloudScript to fire the request.

The said CloudScript is only executed only when a match is found, which in our current game traffic, only happens around less than 10 times (that means < 10 API calls) in 2 mins. We don't think this is the cause due to the small number of calls it takes.

Not that it will help but here are the logs:

cloudscript_executed

{
    "Timestamp": "2021-11-24T08:04:42.8762423Z",
    "EntityLineage": {
        "namespace": "90CED34237D2CE46",
        "title": "78E22"
    },
    "PayloadContentType": "Json",
    "SchemaVersion": "2.0.1",
    "Originator": {
        "Type": "service",
        "Id": "playfab"
    },
    "FullName": {
        "Namespace": "playfab.cloudscript",
        "Name": "cloudscript_executed"
    },
    "Id": "3c1cd969125d4e8c8410b73898759ef8",
    "Payload": {
        "Source": "PlayStreamV2",
        "FunctionName": "matchFoundV2",
        "CloudScriptExecutionResult": {
            "FunctionName": "matchFoundV2",
            "Error": {
                "Error": "CloudScriptExecutionTimeLimitExceeded",
                "Message": "The script execution was terminated after the maximum execution time limit"
            },
            "Revision": 40,
            "ExecutionTimeSeconds": 2.5500415,
            "ProcessorTimeSeconds": 0.002008,
            "MemoryConsumedBytes": 19792,
            "APIRequestsIssued": 0,
            "HttpRequestsIssued": 0
        }
    },
    "Entity": {
        "Type": "title",
        "Id": "78E22"
    },
    "OriginInfo": {}
}

For DeregisterGame API, it is used only in custom games (which are not part of the open beta), so it's being done in the background (around < 20 calls per day).

After careful investigation, we found that we have a lot of player_executed_cloudscript calls in the dedicated server (especially when backfilling match is in process).

  • For every server with incomplete number of players during backfill, we call a CloudScript every 10 seconds until the game is full (getServerBackfillTicket). Question is, does this count to the API limit you are saying?

Also, regarding 100 calls per 2 minutes limit, is this applicable only to the Free Plan? If so, we recently (a day before this question is posted) put our studio plan to pay-as-you-go, but in studio settings, the pay-as-you-go plan is not reflected immediately (and still is right now):

  • Current Plan: Free plan
  • Effective December 1, 2021: Pay as you go plan

Has this change from free plan to pay-as-you-go plan not been always instant?

Is there any way to accelerate this process? Does having a pay-as-you-go plan solve these possible limit problems? We'll definitely make some changes to the frequency of the calls for cost efficiency but now that the game is live, will making our account pay-as-you-go help temporarily solve the throttling issues?

Thank you for taking time to help.

1 comment
10 |1200

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Paolo Sy avatar image Paolo Sy commented ·

I apologize for the spam; I haven't used this forum ever and the replying is confusing.

Mods, kindly remove this post as it is already in the comments in a previous answer. Thank you!

0 Likes 0 ·

Write an Answer

Hint: Notify or tag a user in this post by typing @username.

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.