As request in the slack channel...
We have multiple clients doing auto gameplay/test, all day and night long. Suddenly clients stop because of a script execution time limit exceeded exception. The script and handlers don't do anything special under any circumstances, so they should be more or less deterministic.
The title ID in question is 4D83. The handler is checkForReward
Any ideas what can cause the long execution time?
Btw, the clients in question are running for approximately 3 or 4 hours before this issue comes up. They more or less play the entire game, simulate disconnects, performs re-login, purchases items, consumes items, etc.. They do a lot of stuff and call a lot of server cloud scripts, but there shouldn't be any "unrealistic" bursts sent..
Looking at the behavior, it's all the inventory operations. What's happening is that each grant is adding an item which your client is then using to get the reward. On the backend, each item that is consumed gets tagged for cleanup, so it's technically in the inventory for the player still, just marked for removal (it can't be used in any way, it's just waiting on our "garbage collector", so to speak). Since you're letting the client get a reward item every minute, that's stacking up between cleanup operations. I'm actually surprised that you don't run into it for 3-4 hours, but I can see where that could technically be possible.
The inventory service will be getting an overhaul soon to optimize some of its performance, but for now, I would have to recommend putting more of a delay in the automated testing process - and if the client code can actually do inventory grants as frequently as the test code does, it may be worth re-evaluating how you want to build this part of the logic. For example, since this is a timed reward, why not just use a regenerating currency? You could set it to a max of 1, and the regeneration period to be how often you want to let players get a reward, max. That would effectively eliminate everything but the granting of the final reward from the Cloud Script code.
But otherwise, I do have a couple of recommendations for optimizations:
1. You're getting the inventory multiple times. It'd be better to just get it once at the start, and then have your internal functions pass back the info on any items added. You could then either pass just that back to the client, or add the new items to the inventory info you got at the start, and pass that back.
2. The GrantableSpin item isn't set as Stackable, so each one will be a distinct item in the player inventory. If the intent is to let the player have as many as he wants and use them when he chooses, I'd make it stackable. If the intent is to just do the spin when the reward is clicked on the client, I'd change the item to be a bundle that expires after a few seconds (the bundle adds the items it comes with in that case, and then is cleaned up). In either case though, I would move the rewards table from being in code (the SpinData) to be drop tables (random result tables) in the Catalog.
>> Looking at the behavior, it's all the inventory operations. What's happening is that each grant is adding an item which your client is then using to get the reward. On the backend, each item that is consumed gets tagged for cleanup, so it's technically in the inventory for the player still, just marked for removal (it can't be used in any way, it's just waiting on our "garbage collector", so to speak). Since you're letting the client get a reward item every minute, that's stacking up between cleanup operations. I'm actually surprised that you don't run into it for 3-4 hours, but I can see where that could technically be possible.
Ok, this makes sense to me.
>> The inventory service will be getting an overhaul soon to optimize some of its performance, but for now, I would have to recommend putting more of a delay in the automated testing process - and if the client code can actually do inventory grants as frequently as the test code does, it may be worth re-evaluating how you want to build this part of the logic. For example, since this is a timed reward, why not just use a regenerating currency? You could set it to a max of 1, and the regeneration period to be how often you want to let players get a reward, max. That would effectively eliminate everything but the granting of the final reward from the Cloud Script code.
We will increase the test delay here. In production such a item would be granted only once a day. Only one grantable spin can be available at once, but internally we use this system for all other possible grantable items too... But I still could use a regenerating currency and simply grant the spin item from the server once currency reached > 0 again... (I could use the same checkForRewards handler and instead of keeping track of time manually use the currency and grant the item from there)....
>> 1. You're getting the inventory multiple times. It'd be better to just get it once at the start, and then have your internal functions pass back the info on any items added. You could then either pass just that back to the client, or add the new items to the inventory info you got at the start, and pass that back.
Make sense, but, there reason we implemented it this way is when a user signs in on multiple devices and consumes items on one device. We needed a way to immediately reflect the inventory change on all devices...
>> 2. The GrantableSpin item isn't set as Stackable, so each one will be a distinct item in the player inventory. If the intent is to let the player have as many as he wants and use them when he chooses, I'd make it stackable. If the intent is to just do the spin when the reward is clicked on the client, I'd change the item to be a bundle that expires after a few seconds (the bundle adds the items it comes with in that case, and then is cleaned up). In either case though, I would move the rewards table from being in code (the SpinData) to be drop tables (random result tables) in the Catalog.
The SpinData is hardcoded at the moment, but the actually data will be computed on every spin depending on various factors like user rank, money left, amount of spins performed, owned items, time of day, etc... (I'm not used to drop tables yet, will take a look at it, but I'm not sure if this will be that flexible then)
Thanks a lot for the detailed answer and suggestions, really appreciated.
Understood on the multiple device front. We'll be updating the session service later this year in order to provide a way to revoke sessions (so that you can ensure they can only be on one device at a time). It's certainly the case that a player with two devices can consume items on one while letting the other think it's still there, if you're not using Cloud Script to "use" the item. Just highlighting that the inventory reads are part of the issue, so there may be a way to redesign around that (depending on when/how items are "used").
Thanks for the additional context - that's very helpful!
2 People are following this question.
