question

bkiefer1992 avatar image
bkiefer1992 asked

Server Pending Heartbeat, Terminated, Continuously

We have a multiplayer server up and running for our turn-based multiplayer game (Unity). Players are able to matchmake and have a server spun up for them. When the game's finished, the process will end and the server will shutdown and spin another one up.

Occasionally, the PlayFab server machine running our containers will get into a state where it will continuously be in the 'Pending Heartbeat' state, followed quickly by 'Terminated'. This happens continuously with a phase of about four seconds.

Clicking 'Download logs' produces an empty zip file. Clicking 'Connect' shows a dialog with the message, There was an error while loading data from the server. Please reload the page to retry. The local debug tool seems to run this with no apparent errors, although I've only run it as a process and not as a container (a container is used for the live server).

Our title ID is 9EBFC and the server on which this behaviour is currently happening is: vmss:EastUs:9EBFC_07f1670f-f746-4034-9956-21848477a59e:e178df81-55d8-4f0a-ac62-352228e84d65

Could you please give advice on how to debug this further or fix the issue so that the server doesn't occasionally fall into this state? Thanks very much!

multiplayer
10 |1200
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

1 Answer

·
Xiao Zha avatar image
Xiao Zha answered

We can’t reproduce your issue. You may try using your game servers with LocalMultiplayerAgent and test your game server in container mode before deploying it. Pending heartbeat means something wrong with GSDK integration, so testing with LocalMultiplayerAgent will point this out.

4 comments
10 |1200
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

bkiefer1992 avatar image bkiefer1992 commented ·

Thanks for getting back to me Xiao,

I'll set up the LocalMultiplayerAgent in container mode and test it.

0 Likes 0 ·
bkiefer1992 avatar image bkiefer1992 commented ·

I have set up LocalMutliplayerAgent in container mode. Testing our game in container mode didn't output any errors. Attached you'll find the output from the local multiplayer agent, as well as the console log.

The server that was continuously pending heartbeat and then failing has been closed. If I post the ID of the next server that breaks (happens every week or so) could one of you guys take a look and see if there's anything out of ordinary? Perhaps its running out of memory and closing the Docker container? Seems like if that were the case, it'd spin up a new container.

Thank you for your time!
lmaoutput.txt
pf-consolelogs.txt

0 Likes 0 ·
lmaoutput.txt (88.5 KiB)
pf-consolelogs.txt (59.0 KiB)
Xiao Zha avatar image Xiao Zha bkiefer1992 commented ·

Is this log message: "BK: PlayFabServer: Entering state FlagForShutdown\r\n" generated by the function you implemented? If so, you may be able to find out why the server is shut down by analyzing why the log was generate. Also, you can try using -nographics in the start command.

0 Likes 0 ·
bkiefer1992 avatar image bkiefer1992 Xiao Zha commented ·

The log in which "BK: PlayFabServer: Entering state FlagForShutdown\r\n" was written was that of the LocalMultiplayerAgent run locally in container mode. This was an example of the server running successfully with no outputted errors (It simply timed out after waiting for players to connect, as was expected).

I was unable to download the log from the actual server because clicking the 'Download logs' button on PlayFab yielded an empty zip file. I was also unable to remote into the machine hosting the continuously restarting servers. That lead me to believe that there was a problem with the host machine, since it seems like the host machine should be able to be remoted into even when its containers aren't working correctly.

Fortunately, the error hasn't recurred for about a month now. I don't think further assistance is required.

Thank you very much for your time Xiao.

0 Likes 0 ·

Write an Answer

Hint: Notify or tag a user in this post by typing @username.

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.