Hello
(Please bare with me the long dull opening)
We are in functional spec stage of design our coming online game. We have done many proof of concept coding of using Playfab Server/Client API to achieve game play requirements. So the next step for us is to create testing plans before soft launch. My work is focusing only on technical testing part, so let's ignore any game feature / gameplay related testing in the following discussion.
For client side it's kind straightforward, we mostly do compatibility testing to ensure our mobile game client can run on as many devices as possible. We mostly use TestDroid (now called Bitbar) for global market and Testin.cn for China-specific Android phones (half volume of Android phones used in China). Anyone can find dozen of similar testing services, not only I mentioned above, on the web. In our past experiences above 85% pass result should be good enough.
In past good old days we have to create game server by ourselves, so the question is always on scalability. Testing always involves simulated pressure testing of game server, trying to get some ball park figure how server performed when you have 5000 ccu, 10000 ccu, or 100000 ccu (ccu=concurrent users), etc. We wrote test program to randomly issue 100000 connection request to servers, made them randomly teleport/moving on various of maps, doing stuff, etc. (Actually, we gave free lunch treat for any programmer who found scalability glitch on server :)
This is our first time to use Playfab, so I try to research what kind of scalability testings other people normally do for their Playfab games before 10000 people downloading their app and open the app around same time. However I came almost nothing. Call me old-schooled or terribly paranoid, but I wonder if Playfab offers any way to make simulated testing, or any best practice to allow us to create scenario for scalability testing? On mobile device it's probably not possible to send 10000 simulated registration requests since it might involving same IMEI or only one unique AndroidID, etc.
In our case, what we want to find out the most is if we have 100000 ccu, each client calls some cloudscript function around same time, and this cloudscript function needs to call some external web service to get some data back (like usual POST with 1K JSON data result), so there would be 100000 HTTP Post call to our other web service. Before we worries the scalibility issue of our web service, would that kind of action paralyze Playfab cloudscript even before sending HTTP request out?
Please advise what's the best practice to do scalibility testing, any tip/suggestions would be great!
Best regards,
Charles