I can't see retention in experiment scorecard

Question

question

mariuszsykut asked Jun 27, '23

I can't see retention in experiment scorecard

I'm running an experiment for audience based on segment. I can see the segment has already 1000+ users, but experiment scorecard shows only 9 users. The flow in my game start is: - player is logged to Playfab - player sets statistic on which the segment is based (I didn't want all users to join test, only users from particular game version, which sets mentioned statistic) - experiments list is fetched from Playfab - I can see that user joins experiment (900+ users joined experiment - I can see it using another segment)

Question is - why scorecard shows only 9 users?

analytics

10 |1200

Attachments: Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Answer 1 · 2023-06-29T09:47:09Z

Xiao Zha answered Jun 29, '23

I cannot reproduce your issue. Since the scorecard is canceled after a day, it is possible that the scorecard only counts data for that day, and the same may be true for player counts. Therefore, the scorecard for this one experiment is not reliable, and we suggest that you create a new Experiment to see if the problem persists.

10 |1200

Attachments: Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Answer 2 · 2023-06-27T09:29:13Z

Xiao Zha answered Jun 27, '23

If more than 900 users joined the experiment, but you only see 9 users in the Retention metric, could you share your Scorecard with us? Besides, you’ve mentioned “I can see it using another segment”, could you please share more details about how you set up this segment? Additionally, the Retention metric reflects the percentage of users (not all players) who are logged in N days after seeing the experiment.

1

10 |1200

Attachments: Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

mariuszsykut commented · Jun 27, 2023 at 02:04 PM

Sure, this is how my scorecard looks: https://pasteboard.co/vslZzeieZfiI.png The experiment is still running, but scored status is "Canceled" for some reason.

Experiment audience is based on segment using statistic "BasicExperimentEnabled": https://pasteboard.co/z7gqOr6ikCik.png Client sets it "1" for all new players on particular game version. After the statistic is set, experiments list is fetched. Because player joined segment "Players trying join Basic Fixes Experiment", the experiment is fetched for him. When it happens client sets another statistic - "BasicFixesVariant" - with value 1 or 2 (depending on variant). And using this statistic I made another segment, one to check how many players actually joined experiment, today it has 1070 players: https://pasteboard.co/ELJ1TzsRjCyY.png

0 ·

question