Mutliplayer 2.0 build fails deployment with NoServerHeartbeat, runs fine under MockVmAgent
After a pretty involved process of setting up the GSDK in my UE4 C++ project, I verified that the standalone server runs correctly within MockVmAgent (although not with Docker, since the Setup.ps1 script errors out for me), with heart-beats and proper termination.
However, when I try to deploy a build to production, after a very long time spent "Deploying" (>2 hours), the deployment fails, and I see NoServerHeartbeat as my build's health status.
I was trying to avoid using the legacy multiplayer hosting, but this is quite a blocker. The same build works fine using the legacy system, and clients can match-make and connect to it.
Does anyone have pointers on what could be wrong, or how I could debug this further?
FYI My title ID is BEB5 and my build name(s) is MatchRoyale.
What's the error when you locally testing with Docker? It's highly recommended to verify containerization before deploying a build.
Question how did you zip or packaged your unreal server? I can't seem to pass after Waiting for heartbeats from the game server..... and my logs say they can't load engine libs at path ../../../ when they are clearly there even in the output folder
I use the unreal launcher to build a dedicated server build, and it gets placed in Saved/StagedBuilds. I just zip up the whole directory then (e.g. WindowsServer) and upload that (to the legacy multiplayer system).
Best Answer
For UE4, our guidance is to include the set of dlls that are needed by your server at runtime in the build's assets. I've included some steps below that can help you determine the set of dlls you need.
1. Download ListDlls: https://docs.microsoft.com/en-us/sysinternals/downloads/listdlls
2. Run the server exe that you wish to containerize on your local machine
3. Run listdlls exe pointing at your game. Example: listdlls my_game
4. For the dlls that are not in your exe folder, save the list of dlls. These are the system dlls used by your game.
5. Download the playfab multiplayer docker container (the mock agent setup.ps1 script will do it for you)
6. Start a command line session in a new docker container. The highlighted part will need to be the tag of the docker image downloaded in step 5: docker run -it mcr.microsoft.com/playfab/multiplayer:wsc-10.0.17134.285 cmd
7. Look for the system dlls compiled in step 4. Any that are missing need to be included in the game folder. Let the Playfab team know if you think they should be included in the standard playfab server image.
Thanks for the answer! It was a bit complicated, but I did manage to find all the DLLs that are used by the server binary, and not present in the docker container.
I included them in the build, but unfortunately it still doesn't run. Running the server produces this output:
I'm not sure that'a dll error, though I must admit I'm not very familiar with the environment. Could these runtimes be installed in the server image, or is there any way I can provide them in my build?
Of note is that the legacy EC2 server images run fine, so maybe you can check how those are set up and sync with the new server images?
I'm sorry to be so much of a bother, but hopefully working through this will make the experience smoother for future UE4 developers as well.
I was working through this with another customer recently. They were eventually got it working and provided this guidance:
I added all the dlls that the server was using into the bin directory. Unreal has a small executable that you usually run to start the server or game which then executes another much bigger exe in the bin directory that contains the actual game code. Starting the server using the small executable didn't work for some reason in a container (the server didn't start) but if did a spawned a powershell prompt inside the container and ran the bigger executable directly it started working. Now I've placed a batch file in c:/assets and run that as the start up script which cds into the bin directory and runs the game exe.
Thanks for the ongoing support, this team has one of the most competent and supportive customer support experiences!
Unfortunately, since trying this out didn't work either, my management has decided that we should just use the Legacy multiplayer system until the new one is more stable (i.e. has a working UE4 dedicated server example project / tutorial).
Is there a way I could subscribe to updates about the new system (but especially about UE4 support) so that I can give it another try later this year?
Ok I managed to get Docker running, and I discovered that my server wasn't running at all in the container.
This led me to believe this has something to do with the different OS versions, and indeed, upon RDC-ing into the running VM and attempting to run the server binary manually, I get a pop-up informing me that Visual C++ and DirectX are not provided.
I tried installing them by running the UE4PrereqSetup tool, but this also fails in the server environment (even after installing some stuff like .Net 3.5 manually).
In conclusion, the container image provided is not able to run an Unreal Engine dedicated server at the time. This unfortunately means I'll have to rely on legacy multiplayer builds only, as those VMs seem to run the server fine.
@jayzuo Do my conclusions here make sense? Is there a channel I can follow for possible future improvements to this situation? I really wanted to play nice and use the new multiplayer system (especially since the new matchmaker has some features that would make life easier).
When I was testing on "competing offerings", I had to create an install step to install msvcrt and directX on the machine because it wasn't included by default on a windows server. Once those were setup, my UE4 dedicated server had no issues running. I was not using docker containers though.
That sounds promising, do you have any details about this install step and how you set it up?
I was able to solve this problem by adding this line to the Build.cs file of my server module:
RuntimeDependencies.Add("$(TargetOutputDir)/xinput1_3.dll", "$(EngineDir)/Binaries/ThirdParty/AppLocalDependencies/Win64/DirectX/xinput1_3.dll", StagedFileType.SystemNonUFS);
This basically just copies the xinput dll from a folder in the engine to alongside the output binary.
The ListDlls steps above worked for me, but gave me a larger list of dlls than were actually required to run the server.
I'd also suggest when building to specify the -nobootstrapexe flag (at least if you are using RunUAT) so that it doesn't include the bootstrapper which is hardcoded to look for certain dlls and will show a message box if they don't exist, which isn't very server friendly.
For anyone still struggling with this you also need to change all references of vcruntime140 in your Server.build.cs script to vcruntime140_1. It looks like the dll requirements have change and the GSDK integration guide hasn't yet been updated.
Has anyone managed to get the server working with UE4? I've been trying for days and I always get the error "TooManyRestarts" and "Unhealthy".
