There seems to be a very serious issue with how treatment assignment is being handled.
My client is android only. It uses react native with the javascript sdk.
I want to do an experiment which changes the levelling system in my game.
Most of the levelling is done on the client but some assignment is done on the server.
Because we cannot access treatment variables in cloudscript, we copy them into an object in title data called experimentData. ExperimentData contains the variantId and a copy of the data so we can use it in the cloud script. This part works ok.
My client app does loginWithAndroidDeviceId on startup and it gets the Treatment Assignments in the response.
1.
The problem is, if I run login multiple times, I get different results.
Below I have grepped for variants
07-26 20:14:23.855 31911 32424 I ReactNativeJS: variants: [ '956421' ] }
07-26 20:15:43.606 3687 4156 I ReactNativeJS: variants: [ '956421' ] }
07-26 20:16:45.052 8492 8963 I ReactNativeJS: variants: [ 'e67085', 'ac6860' ] }
07-26 20:17:35.387 12252 12689 I ReactNativeJS: variants: [ '956421' ] }
07-26 20:19:37.253 16050 16471 I ReactNativeJS: variants: [ '956421' ] }
07-26 20:20:28.883 19153 19838 I ReactNativeJS: variants: [ '956421' ] }
07-26 20:21:17.487 22675 23154 I ReactNativeJS: 'ConfigurationManager/SET_TREATMENTS', { variables: [], variants: [] }
07-26 20:22:04.276 26001 26494 I ReactNativeJS: variants: [ '956421' ] }
07-26 20:22:53.730 29488 30292 I ReactNativeJS: variants: [ '956421' ] }
07-26 20:23:44.991 798 1314 I ReactNativeJS: variants: [ '956421' ] }
07-26 20:24:32.337 4617 5087 I ReactNativeJS: variants: [ '956421' ] }
07-26 20:25:19.859 8900 9359 I ReactNativeJS: 'ConfigurationManager/SET_TREATMENTS', { variables: [], variants: [] }
07-26 20:26:08.908 12285 12720 I ReactNativeJS: variants: [ '956421' ] }
07-26 20:29:26.659 22070 22598 I ReactNativeJS: variants: [ 'e67085', 'ac6860' ] }
07-26 20:30:10.733 25128 25525 I ReactNativeJS: variants: [ '956421' ] }
07-26 20:31:01.045 28379 29089 I ReactNativeJS: 'ConfigurationManager/SET_TREATMENTS', { variables: [], variants: [] }
07-26 20:31:48.017 32122 32603 I ReactNativeJS: 'ConfigurationManager/SET_TREATMENTS', { variables: [], variants: [] }
07-26 20:32:37.660 3181 3908 I ReactNativeJS: variants: [ '956421' ] }
07-26 20:33:27.440 7719 8295 I ReactNativeJS: variants: [ '956421' ] }
07-26 20:34:13.635 11716 12130 I ReactNativeJS: 'ConfigurationManager/SET_TREATMENTS', { variables: [], variants: [] }
07-26 20:35:00.258 15099 15499 I ReactNativeJS: 'ConfigurationManager/SET_TREATMENTS', { variables: [], variants: [] }
07-26 20:35:50.281 18375 19116 I ReactNativeJS: variants: [ '956421' ] }
07-26 20:36:38.223 22154 22858 I ReactNativeJS: variants: [ '956421' ] }
07-26 20:37:26.136 25997 26535 I ReactNativeJS: variants: [ '956421' ] }
07-26 20:38:13.811 29946 30596 I ReactNativeJS: variants: [ 'e67085', 'ac6860' ] }
07-26 20:39:00.842 877 1579 I ReactNativeJS: variants: [ '956421' ] }
07-26 20:41:28.005 12222 12753 I ReactNativeJS: variants: [ '956421' ] }
07-26 20:45:39.967 27728 28265 I ReactNativeJS: variants: [ '956421' ] }
As you can see, successive logins are giving me different results. Sometimes, I get variant
[956421], sometimes I get variants ['e67085', 'ac6860'] and sometimes I get no variants at all.
2.
This problem also happens in cloudscript. I have a cloudscript function that will return different data depending on whether the user has variant 956421 or not. This is also returning inconsistent results.
3.
Variant e67085 and ac6860 are from experiments that I stopped several hours ago. So it shouldn't be possible for these to appear. I considered that this user might be in a bad state so I unlinked my device from the user account, and restarted the app (thus creating a new user)
07-26 20:46:39.952 31732 32214 I ReactNativeJS: variants: [ '1fe66d', 'ac6860' ] }
07-26 20:47:29.525 2979 3562 I ReactNativeJS: 'ConfigurationManager/SET_TREATMENTS', { variables: [], variants: [ 'ce8f2c' ] }
07-26 20:48:16.638 6745 7380 I ReactNativeJS: 'ConfigurationManager/SET_TREATMENTS', { variables: [], variants: [ 'ce8f2c' ] }
07-26 20:49:04.518 10743 11221 I ReactNativeJS: 'ConfigurationManager/SET_TREATMENTS', { variables: [], variants: [] }
This user ends up in a similar situation. The very first run for this user included 2 variant ids that are in the experiments that were stopped earlier in the day.
So this is a case where the experiment was stopped before the user account was created, but the use is still entered into that experiment.
4.
I have more experiments that are stopped but my users are never entered into them which leads me to believe this may be a localised problem. I deleted all stopped experiments, unlinked my device and tried again.
07-26 21:19:40.816 26453 26919 I ReactNativeJS: variants: [ '956421' ] }
07-26 21:20:31.719 29756 30306 I ReactNativeJS: 'ConfigurationManager/SET_TREATMENTS', { variables: [], variants: [] }
07-26 21:21:21.013 593 1156 I ReactNativeJS: 'ConfigurationManager/SET_TREATMENTS', { variables: [], variants: [] }
07-26 21:22:08.590 4432 4825 I ReactNativeJS: variants: [ '956421' ] }
07-26 21:22:54.159 8761 9270 I ReactNativeJS: variants: [ '956421' ] }
07-26 21:23:46.850 12420 12889 I ReactNativeJS: variants: [ '956421' ] }
07-26 21:24:33.317 15959 16515 I ReactNativeJS: variants: [ '956421' ] }
07-26 21:25:21.340 19455 19860 I ReactNativeJS: 'ConfigurationManager/SET_TREATMENTS', { variables: [], variants: [] }
07-26 21:26:09.052 22688 23106 I ReactNativeJS: 'ConfigurationManager/SET_TREATMENTS', { variables: [], variants: [] }
07-26 21:26:57.996 25897 26323 I ReactNativeJS: variants: [ '956421' ] }
07-26 21:27:41.615 29070 29489 I ReactNativeJS: variants: [ '956421' ] }
07-26 21:28:31.541 32384 396 I ReactNativeJS: variants: [ 'e67085', 'aef87' ] }
Here again we can see the same behaviour even though I have only 1 experiment running, and have deleted all stopped, draft and scheduled experiments.
5.
I have a separate playfab app for production and found this user who had the same behaviour:
Logins a couple hours apart, where the variant was applied inconsistently.
UserId, Variants, EventTime
19412E5CB75769F6, ["7490f","695981"], 2020-07-21 00:15:20.307
19412E5CB75769F6, [ ], 2020-07-21 03:43:32.980
19412E5CB75769F6, ["7490f","695981"], 2020-07-21 07:30:52.837
