question

agileen2002 avatar image
agileen2002 asked

Best practice of Playfab online game technical testing ?

Hello

(Please bare with me the long dull opening)

We are in functional spec stage of design our coming online game. We have done many proof of concept coding of using Playfab Server/Client API to achieve game play requirements. So the next step for us is to create testing plans before soft launch. My work is focusing only on technical testing part, so let's ignore any game feature / gameplay related testing in the following discussion.

For client side it's kind straightforward, we mostly do compatibility testing to ensure our mobile game client can run on as many devices as possible. We mostly use TestDroid (now called Bitbar) for global market and Testin.cn for China-specific Android phones (half volume of Android phones used in China). Anyone can find dozen of similar testing services, not only I mentioned above, on the web. In our past experiences above 85% pass result should be good enough.

In past good old days we have to create game server by ourselves, so the question is always on scalability. Testing always involves simulated pressure testing of game server, trying to get some ball park figure how server performed when you have 5000 ccu, 10000 ccu, or 100000 ccu (ccu=concurrent users), etc. We wrote test program to randomly issue 100000 connection request to servers, made them randomly teleport/moving on various of maps, doing stuff, etc. (Actually, we gave free lunch treat for any programmer who found scalability glitch on server :)

This is our first time to use Playfab, so I try to research what kind of scalability testings other people normally do for their Playfab games before 10000 people downloading their app and open the app around same time. However I came almost nothing. Call me old-schooled or terribly paranoid, but I wonder if Playfab offers any way to make simulated testing, or any best practice to allow us to create scenario for scalability testing? On mobile device it's probably not possible to send 10000 simulated registration requests since it might involving same IMEI or only one unique AndroidID, etc.

In our case, what we want to find out the most is if we have 100000 ccu, each client calls some cloudscript function around same time, and this cloudscript function needs to call some external web service to get some data back (like usual POST with 1K JSON data result), so there would be 100000 HTTP Post call to our other web service. Before we worries the scalibility issue of our web service, would that kind of action paralyze Playfab cloudscript even before sending HTTP request out?

Please advise what's the best practice to do scalibility testing, any tip/suggestions would be great!

Best regards,

Charles

apis
10 |1200

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

1 Answer

·
brendan avatar image
brendan answered

The simple answer is that developers don't do scale testing on PlayFab, as there's no point in it. We currently have close to 80 million monthly active users (MAU) across more than 1,000 live games. Our systems are load tested on a daily basis, and our CCU level peaks in the millions.

But that said, if you have a corporate requirement to do some kind of load testing for purposes of providing feedback to your executive team, we can work with you on that as part of an enterprise agreement. If you have a look at our technical whitepaper (http://s3-us-west-2.amazonaws.com/api-playfab-com-craft-files/FileAssets/playfabtechnicalwhitepaper_2016.06.18.pdf), the 1M CCU load test we did for a major publisher is an example of something we could coordinate with you on, given that you'll be covering the costs for that testing. If that's something you'd like to discuss with us in more depth, feel free to reach out to us at devrel@playfab.com.

1 comment
10 |1200

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

agileen2002 avatar image agileen2002 commented ·

Thanks Brendan, that actually clarify the doubts I have:)

0 Likes 0 ·

Write an Answer

Hint: Notify or tag a user in this post by typing @username.

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.