question

Seth avatar image
Seth asked

A-B tests

So I'm trying to figure out how I would create a good ab-test environment which I can somewhat control from the Game Manager.

I'm thinking about having a cloudscript handler at login which just assigns a random nr (0-100) to the user statistics data. This way I could segment the a percentage of all players with Playstream segmentation (if statistics value "AB-Id less than...)

But then I still don't understand how I would be able to send some kind of data to the client to know that it now is a part of an AB-test. Simplest solution would be to just send a key value pair ( eg. { "Funnel_flow": "B" }) that is then handled and interpreted by the client. 

Using the segments would be awesome as I then can get the statistics out segmenting the same AB groups.

Am I going to wrong way about this? Is there any other way I could try to push information to the client (preferable whenever I start a new AB-test from gamemanager) ?

10 |1200

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

brendan avatar image
brendan answered

We'll be providing an A/B feature later this year, but if you wanted to implement one now, I would actually use the PlayFabId modulo N, where N is the granularity you want to be able to have (so, if you want to be able to fine-tune your experiment groupings at the single-digit percentage level, 100). For the data differences, you would set all of them up in title data as you describe, and then either let the client choose the correct one based upon the experiment group they're in, or else use Cloud Script to control which data is sent to the player.

10 |1200

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Seth avatar image
Seth answered

Using the PlayFabId as the base for selecting out groups is a good idea, problem is that I still want to use the segmentation for triggering AB-tests. This way I can then later on use segmentation to see all the statistics of the seperate testgroups. Unfortunately today there are no other way of using segmentation that I can thing of. 

Using "statistics lower than" will get my group of players which has a special funnel flow or special store etc.

I'm very close to do this and my theory was this: As soon as I have the statistic value set I can then group people with that. They will immediately trigger a cloudscript as entering the segment which in turn writes what special AB-test they are active in (eg "Funnel:b", "Gamemode:B", "Store:A" etc) in a ReadOnly user data. Problem I now stumble upon is that there is no way to send an argument to cloudscripts when Segments are triggered. So I just cannot tell the cloudscript to register the user to a certain test and in a certain group. 

Only solution right now is to have one cloudscript handler for each AB-test I would like to have but it feels really annoying. So, a request coming from this is to be able to set simple arguments to the cloudscript being triggered in Segment enter and leaving actions. 

Other than that I might try some simple AB-test methods but wait for the big implementation for your update later on this year. 

Thanks for the answer. 

10 |1200

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

brendan avatar image
brendan answered

Aha - you're exploring PlayStream, then? Great! We'll have more docs available on it posted to the site shortly, but one way you could do this is by using the "context", which contains all the event info for the call.

Again, we'll have that posted shortly, but the way this works is that there's a new parameter you can use in the inputs to your Cloud Script handler. So now, instead of:

handlers.doThing = function (args)

You would use:

handlers.doThing = function (args, context)

We're finalizing a few things on the docs now, so please keep your eyes on that page for the update (that way, we don't risk conflicting info here). We should have that available on the site tomorrow.

So, one thing you could do would be to use a custom event (using WriteEvent - again, that'll have all the info on the site as of tomorrow), and use the data you pass in via that event in your Cloud Script.

Alternately, the order of operations in the actions defined for the Segment enter/exit is guaranteed, so you could also set a statistic as the first action, then call the Cloud Script second and use the statistic you just set in the logic.

10 |1200

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Write an Answer

Hint: Notify or tag a user in this post by typing @username.

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.