We're looking for a way to compare our different A/B-Test segments regarding various metrics (revenue, retention, etc.).
The only option we have found so far is the Daily AB Test Kpi Report - however this isn't practical for us since data from a single day just doesn't include enough players/purchases to be statistically meaningful.
Are there other options, that would let us compare data from a longer period of time?
What's the "usual" way to evaluate playfab A/B-tests?
Hi Larissa, as far as I am aware, the daily report is currently the only official entrance that PlayFab offers to review results. Although you could gather every single day’s data into one data sheet and evaluate it, which is probably more statistically meaningful but might be a hassle for you. However, I may have come up with a option for you to consider. Let’s suppose you are doing a A/B test on item prices, players from different buckets will purchase items with different prices(suppose that players are purchasing items with virtual currency). Every purchase made will generate a play stream event, in this case, the event “player_vc_item_purchased” will be generated. Now, with the new feature Explorer PlayFab is offering, you can easily query the data from any period of time for your evaluation purpose. Please have a try to see if it meets your needs, any issues let us know:)
Right, combining the daily data would be quite time-consuming so we'd rather avoid that. Would be really cool if a monthly report would be added in the future!
Also, thanks for the suggestion! :)
I've checked out the explorer, but couldn't find a way to get the AB-test-bucket associated with the events - am I missing something or is it just not possible?
I can see that for some tests like item prices one could use other event-properties to indirectly filter/group by bucket.
However our A/B-tests include various forms of different client behaviour that don't result in any unique playstream-event differences (e.g. one graphic is exchanged for another). So we would really need to find out the bucket with the query then.
Hi Larissa, about this monthly report, you can make a feature request here: https://community.playfab.com/spaces/24/index.html. It seems that the Explorer is unable to get the AB-test-bucket associated with the events. Yes, for some tests like item prices you can always find a way to indirectly group them by bucket. About the tests that really needs bucket filtered, can you describe it more specific so that maybe we can come up with a workaround together. Or you can make a feature request about filtering buckets with query too:)
3 People are following this question.
