question

>> My questions is as we understand from our daily player count, even though we would like to improve our new users' stats, we see retention values for existing users also, does it mean that Day 1 retention for existing users are the retentions after the day experiment started?

If you checked the “Run across all title players“ when you create the Experiment, the Experiment will affect all players. Day 1 is the day after the given day(Day 0), it’s not related to Experiment.

>> If my above assumption is right, what is the proper way for making tests that effects only newly created users to improve retention and engagement without intervention of already engaged users?

PlayFab provides the Thirty Day Retention report and the Thirty Day New User Retention report. As the documentation said, the Thirty Day Retention report illustrates how many of your players came back to the game within a 30-day period. And the Thirty Day New User Retention report tracks the new users’ retention over a 30-day period. You can check the Thirty Day New User Retention report to check the new users’ retention rate.

If you only want to run the Experience for new users, you need to create a Segment for new users first. For example, assuming the new users mean the players who registered within 1 month, the filer condition of the Segment should be “First login is less than or equal to 43,200(minutes)”. Then you need to check the “Select player segment” in the [Edit Experiment] page and chose the new users’ Segment.