question

Francois Boucher-Genesse avatar image
Francois Boucher-Genesse asked

Measuring the success of an experiments with a custom score

Hello,

We'd like to do A/B testing with PlayFab Experiments for IAP related features. We handle all purchases ourselves under a subscription model, so PlayFab isn't aware of purchases at the moment. This forum thread seems to indicate that we can't make PlayFab aware of purchases and have it send player_realmoney_purchase events.

This means that the outcomes of the experiments we care about, activations and revenue, will not have anything in the Experiments Scorecards.

My understanding is that statistical tools like the p-value analysis provided by PlayFab will only work with the 4 metrics backed in scorecards, and that we cannot send additional custom metrics for PlayFab to track. Is this correct?

If so, I think this means we will need to rely on our own telemetry to determine ourselves the outcome of experiments. But I'd love to be told I'm wrong here :)

Thanks

10 |1200
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

2 Answers

·
Sarah Zhang avatar image
Sarah Zhang answered

Yes, as you said, you can’t add a custom metric to the Experiment Scorecards. And except for the first time, no event will be generated when the subscription is charged. If you want to add a metric that can track subscription fees in the experiment, you can try to add a feature request for it. Currently, you need to analyze it using you own method.

10 |1200
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Francois Boucher-Genesse avatar image
Francois Boucher-Genesse answered
@Sarah Zhang

Understood thanks. The way PlayFab handled subscriptions on iOS wasn't functional yet, so we implemented it ourselves. Being able to feed purchase information to PlayFab in general would be more useful than changing subscriptions for us, but also more flexible for other developers. That already seems on PlayFab's radar.

10 |1200
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Write an Answer

Hint: Notify or tag a user in this post by typing @username.

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.