question

Please correct me if I misunderstood something:

You self-hosted dozens of long-running servers where there could be few players playing in each of them at any time, and, when the number of players in some of the servers is low or even empty, the servers will create backfill tickets to search for additional players, is that the right flow of yours?

If that’s the case, based on the example you provided, you probably hit the “Server backfill ticket creates per minute” limit:

“This limits how frequently a user can become a member of server backfill tickets. A user becomes a member of a backfill ticket when a server creates a ticket with them in it. This limit is per user per queue.”

And, looks like you’re trying to search for only one player at a time, which could potentially hit the limit soon since that would require creating plenty of tickets to search for a large number of players. As a matter of fact, a single server backfill ticket can match with multiple regular tickets (tickets created by players in this case), which means that you can search for multiple players instead of one at a time, to do that, you’ll just need to raise the match size in the Queue Configuration.

Moreover, what exactly did you set for the “Members” property when making the CreateServerBackfillTicket requests? Is it the list of players the current server is having or something else?

BTW, we cannot see the request statistics without your title id, would you please share that?

Title ID: E7560

Each server has max players limit. But it has no low bound. It means, that server will try match more and more players till the max limit (via backfill API). Players can leave server and this is the second reason for backfill.

Of course, there is no limitation for only one player a time. Backfill MM could match more than one at the time, but imagine, that each second one player left server and always there is one player in queue. Server mm logic ant timings are:

00:00:00: Player/players gone. Generate Backfill to get new players with a list of all active players on this server.

00:00:06: check Backfill ticket (6 sec is a timeout to avoid mm check frequency limits)

00:00:06: MM happened. New players added, but some players already left (or added not enough to reach the server's max player limit). Makes a new Backfill ticket with an updated list of players.

00:00:12: check Backfill ticket (6 sec is a timeout to avoid mm check frequency limits) .... etc

I want to reduce client waiting time. Let's why the server in the most has to have a backfill ticket in the waiting state.

I've made a test:

• there is a mm queue "q1" with rule to have one user with "isServer" attr and 3 others.
• there are clients, making mm tickets and checking it each 6 sec. After match happened, it create MM ticket again with new user.
• there are server. It owns the user with "isServer" attr. And cycle with creating backfill ticket with this user in Members and waiting for match. When match happens, it cancel backfill tickets for this user and repeat cycle iteration.
• all users are looged in via custom login id

test case:

• 25 clients
• 3 servers

Here is citation from logs:

[2020-05-15T13:55:58.188582] _create_mm_ticket: creating backfill ticket with user [client_server_id_1]  with entityId [42F24FFDFF7311F2]
[2020-05-15T13:55:58.201873] _create_mm_ticket: creating backfill ticket with user [client_server_id_2]  with entityId [6280B299F73DD395]
[2020-05-15T13:55:58.221893] _create_mm_ticket: creating backfill ticket with user [client_server_id_0]  with entityId [D672636F8DB3CFAF]
[2020-05-15T13:56:10.162567] _create_mm_ticket: creating backfill ticket with user [client_server_id_1]  with entityId [42F24FFDFF7311F2]
[2020-05-15T13:56:10.166058] _create_mm_ticket: creating backfill ticket with user [client_server_id_2]  with entityId [6280B299F73DD395]
[2020-05-15T13:56:10.325918] _create_mm_ticket: creating backfill ticket with user [client_server_id_0]  with entityId [D672636F8DB3CFAF]
[2020-05-15T13:56:21.499047] _create_mm_ticket: creating backfill ticket with user [client_server_id_2]  with entityId [6280B299F73DD395]
[2020-05-15T13:56:21.667310] _create_mm_ticket: creating backfill ticket with user [client_server_id_0]  with entityId [D672636F8DB3CFAF]
[2020-05-15T13:56:22.093076] _create_mm_ticket: creating backfill ticket with user [client_server_id_1]  with entityId [42F24FFDFF7311F2]
[2020-05-15T13:56:32.587632] _create_mm_ticket: creating backfill ticket with user [client_server_id_2]  with entityId [6280B299F73DD395]
[2020-05-15T13:56:32.816504] _create_mm_ticket: creating backfill ticket with user [client_server_id_0]  with entityId [D672636F8DB3CFAF]
[2020-05-15T13:56:33.193930] _create_mm_ticket: creating backfill ticket with user [client_server_id_1]  with entityId [42F24FFDFF7311F2]
[2020-05-15T13:56:44.856854] _create_mm_ticket: creating backfill ticket with user [client_server_id_2]  with entityId [6280B299F73DD395]
[2020-05-15T13:56:44.984166] _create_mm_ticket: creating backfill ticket with user [client_server_id_0]  with entityId [D672636F8DB3CFAF]
[2020-05-15T13:56:45.135035] _create_mm_ticket: creating backfill ticket with user [client_server_id_1]  with entityId [42F24FFDFF7311F2]
[2020-05-15T13:56:49.118708] _create_mm_ticket: creating backfill ticket with user [client_server_id_2]  with entityId [6280B299F73DD395]
[2020-05-15T13:56:49.216336] _create_mm_ticket: creating backfill ticket with user [client_server_id_0]  with entityId [D672636F8DB3CFAF]
2020-05-15 13:56:50,078  Exception in ticket processing src.pf_error.PFException: create_server_backfill_ticket error: {'code': 429, 'status': '429', 'error': 'MatchmakingRateLimitExceeded', 'errorCode': 2054, 'errorMessage': 'The rate of create or join ticket requests for title!E7560 has exceeded the limit and is temporarily throttled.'} 
2020-05-15 13:56:50,184  Exception in ticket processing src.pf_error.PFException: create_server_backfill_ticket error: {'code': 429, 'status': '429', 'error': 'MatchmakingRateLimitExceeded', 'errorCode': 2054, 'errorMessage': 'The rate of create or join ticket requests for title!E7560 has exceeded the limit and is temporarily throttled.'} 
[2020-05-15T13:56:54.976535] _create_mm_ticket: creating backfill ticket with user [client_server_id_2]  with entityId [6280B299F73DD395]
[2020-05-15T13:56:55.104524] _create_mm_ticket: creating backfill ticket with user [client_server_id_0]  with entityId [D672636F8DB3CFAF]
2020-05-15 13:56:55,942  Exception in ticket processing src.pf_error.PFException: create_server_backfill_ticket error: {'code': 429, 'status': '429', 'error': 'MatchmakingRateLimitExceeded', 'errorCode': 2054, 'errorMessage': 'The rate of create or join ticket requests for title!E7560 has exceeded the limit and is temporarily throttled.'} 
2020-05-15 13:56:56,097  Exception in ticket processing src.pf_error.PFException: create_server_backfill_ticket error: {'code': 429, 'status': '429', 'error': 'MatchmakingRateLimitExceeded', 'errorCode': 2054, 'errorMessage': 'The rate of create or join ticket requests for title!E7560 has exceeded the limit and is temporarily throttled.'} 
[2020-05-15T13:56:56.472514] _create_mm_ticket: creating backfill ticket with user [client_server_id_1]  with entityId [42F24FFDFF7311F2]
2020-05-15 13:56:57,476  Exception in ticket processing src.pf_error.PFException: create_server_backfill_ticket error: {'code': 429, 'status': '429', 'error': 'MatchmakingRateLimitExceeded', 'errorCode': 2054, 'errorMessage': 'The rate of create or join ticket requests for title!E7560 has exceeded the limit and is temporarily throttled.'} 
[2020-05-15T13:57:00.950763] _create_mm_ticket: creating backfill ticket with user [client_server_id_2]  with entityId [6280B299F73DD395]
[2020-05-15T13:57:01.317953] _create_mm_ticket: creating backfill ticket with user [client_server_id_0]  with entityId [D672636F8DB3CFAF]
[2020-05-15T13:57:02.403938] _create_mm_ticket: creating backfill ticket with user [client_server_id_1]  with entityId [42F24FFDFF7311F2]
[2020-05-15T13:57:12.039071] _create_mm_ticket: creating backfill ticket with user [client_server_id_2]  with entityId [6280B299F73DD395]
[2020-05-15T13:57:12.505056] _create_mm_ticket: creating backfill ticket with user [client_server_id_0]  with entityId [D672636F8DB3CFAF]

Result: the limit is about 5 backfill tickets per minute with the same user in Members.