Custom server load testing: how?

Question

question

Anton asked Feb 6, '19

Custom server load testing: how?

Hi. Before soft launch, I need to understand how the game scales.

We have a game written purely on c# with unity. Our server logic is divided to META and BATTLE modes that execute as Playfab Custom Servers.

Battle handles some realtime interactions - one battle per server instance (2-20) players. It just relays commands among players.

Meta is for some player progression part. So Meta reads player from Playfab database when player authes, than player rarely executes some command batches during his session. Than once player left, we update player's database entry. We assume we can do 1k-100k players with one server instance.

I know there's a lot of questions about load testing already, but with custom servers its more about AWS load testing. AWS has CloudWatch - a nice tool for monitoring its resources (cpu, network, disk, memory etc)

For example for monitoring cpu for our default t2.medium servers there is a useful api there:

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/burstable-performance-instances-monitoring-cpu-credits.html

First question: so can I get access to CloudWatch API from my Playfab account?

If no, whats are best practices to make load tests on custom servers? I mean how can I understand how many clients can I handle on one t2.medium host instance and/or decide which AWS plan to choose. I can't just measure cpu usage and network traffic with my custom server because there's a hell lot of factors that will make AWS numbers different from what I'd receive.

Second question, why 'Max instances per host' is build setting and not mode setting? In our game the same build operates in 2 modes and eats significantly different resource amounts. So to tell Playfab that there can be 2 META-games per one host and 100 BATTLE-games per one host, I need to upload the same build twice. Also it seems that I cant therefore have 1 META-game and 50 BATTLE-games on one host instance.

Third question, How should we decide about 'Max instances per host' and 'Max players count' for META server? Yes, we actually decided to avoid using CloudScript and make our game operate purely on custom servers.So can we have 'Max players count' in this case insanely large (like 100k)? what limits should be we aware of? network connections count? game istance limits on Playfab API requests? Is there a reason to split our META-server from 1 instance per host X 100k max players count to, say, 100 instances per host X 1000 max players? I mean, if those amounts of players fit cpu and memory limits of our AWS plan.

Fourth question: when I start doing load testing, how to avoid being blocked because of client requests amount from same IP? Without that testing I obviously can't determine my 'Max instances per host' and 'Max players count' for both game modes for my AWS plan.

Custom Game Servers test

1

10 |1200

Attachments: Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Anton commented · Feb 06, 2019 at 08:46 PM

title id5626

0 ·

Answer 1 · 2019-02-06T19:35:15Z

brendan answered Feb 6, '19

If you want to do load testing ahead of launch, and you want access to the full range of AWS features for monitoring, you'll need to do that testing in an AWS account that you own an manage. The issue is that we cannot provide direct access to the tools you mentioned without causing a serious security issue (exposing access to the servers of other titles). That's one of the reasons we're moving to Azure for our newer server hosting - to provide more debugging and monitoring options.

The AMI is a fairly ordinary Windows 2K12 R2 build, though we do have a manager loaded which communicates with our game wrangler (this is what lets us manage spinning up and down instances). It's not very memory or CPU intensive though, so if you target the normal 80% capacity max, you should be fine.

Max instances per host is per build in order to avoid specifically the scenario you describe - where multiple instances with completely different load characteristics want to share a host machine. So yes, if you have a build that operates in different modes with different characteristics, you should really have it loaded twice, as different "builds", as they'll need to run on different hosts.

For the max players for a game mode, the theoretical limit on the number of TCP/IP connections is 16 million. Of course, we haven't tested anywhere near that level of connections, and we certainly wouldn't recommend trying. The maximum request rate for the Server API is 1,000 per 10 seconds, which would likely be the best thing to use in computing how many players you want to allow per server. Bear in mind that the limit is for all players - trying to make many calls on a single player account in a short period would still have performance issues.

Finally, the client request limit is there to prevent any one title from impacting others, due to a bug or other issue causing pathological behavior. There's no "workaround" to bypass it, as that would defeat the purpose of having that limit in the service. But that limit is specifically for API calls to our service, not to your custom game server. If you throttle the rate of your test harness calls (to the Matchmaker, etc.), you would still be able to hit your own server as hard as you want to.

10 |1200

Attachments: Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

question