question

Mircea Pricop avatar image
Mircea Pricop asked

Mutliplayer 2.0 build fails deployment with NoServerHeartbeat, runs fine under MockVmAgent

After a pretty involved process of setting up the GSDK in my UE4 C++ project, I verified that the standalone server runs correctly within MockVmAgent (although not with Docker, since the Setup.ps1 script errors out for me), with heart-beats and proper termination.

However, when I try to deploy a build to production, after a very long time spent "Deploying" (>2 hours), the deployment fails, and I see NoServerHeartbeat as my build's health status.

I was trying to avoid using the legacy multiplayer hosting, but this is quite a blocker. The same build works fine using the legacy system, and clients can match-make and connect to it.

Does anyone have pointers on what could be wrong, or how I could debug this further?

Custom Game Servers
4 comments
10 |1200

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Mircea Pricop avatar image Mircea Pricop commented ·

FYI My title ID is BEB5 and my build name(s) is MatchRoyale.

0 Likes 0 ·
JayZuo avatar image JayZuo ♦ commented ·

What's the error when you locally testing with Docker? It's highly recommended to verify containerization before deploying a build.

0 Likes 0 ·
fiercy avatar image fiercy commented ·

Question how did you zip or packaged your unreal server? I can't seem to pass after Waiting for heartbeats from the game server..... and my logs say they can't load engine libs at path ../../../ when they are clearly there even in the output folder

0 Likes 0 ·
Mircea Pricop avatar image Mircea Pricop fiercy commented ·

I use the unreal launcher to build a dedicated server build, and it gets placed in Saved/StagedBuilds. I just zip up the whole directory then (e.g. WindowsServer) and upload that (to the legacy multiplayer system).

0 Likes 0 ·
Andy avatar image
Andy answered

For UE4, our guidance is to include the set of dlls that are needed by your server at runtime in the build's assets. I've included some steps below that can help you determine the set of dlls you need.

1. Download ListDlls: https://docs.microsoft.com/en-us/sysinternals/downloads/listdlls

2. Run the server exe that you wish to containerize on your local machine

3. Run listdlls exe pointing at your game. Example: listdlls my_game

4. For the dlls that are not in your exe folder, save the list of dlls. These are the system dlls used by your game.

5. Download the playfab multiplayer docker container (the mock agent setup.ps1 script will do it for you)

6. Start a command line session in a new docker container. The highlighted part will need to be the tag of the docker image downloaded in step 5: docker run -it mcr.microsoft.com/playfab/multiplayer:wsc-10.0.17134.285 cmd

7. Look for the system dlls compiled in step 4. Any that are missing need to be included in the game folder. Let the Playfab team know if you think they should be included in the standard playfab server image.

4 comments
10 |1200

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Mircea Pricop avatar image Mircea Pricop commented ·

Thanks for the answer! It was a bit complicated, but I did manage to find all the DLLs that are used by the server binary, and not present in the docker container.

I included them in the build, but unfortunately it still doesn't run. Running the server produces this output:

I'm not sure that'a dll error, though I must admit I'm not very familiar with the environment. Could these runtimes be installed in the server image, or is there any way I can provide them in my build?

Of note is that the legacy EC2 server images run fine, so maybe you can check how those are set up and sync with the new server images?

I'm sorry to be so much of a bother, but hopefully working through this will make the experience smoother for future UE4 developers as well.

0 Likes 0 ·
capture.png (104.8 KiB)
Andy avatar image Andy ♦♦ Mircea Pricop commented ·

I was working through this with another customer recently. They were eventually got it working and provided this guidance:

I added all the dlls that the server was using into the bin directory. Unreal has a small executable that you usually run to start the server or game which then executes another much bigger exe in the bin directory that contains the actual game code. Starting the server using the small executable didn't work for some reason in a container (the server didn't start) but if did a spawned a powershell prompt inside the container and ran the bigger executable directly it started working. Now I've placed a batch file in c:/assets and run that as the start up script which cds into the bin directory and runs the game exe.

1 Like 1 ·
Mircea Pricop avatar image Mircea Pricop Andy ♦♦ commented ·

Thanks for the ongoing support, this team has one of the most competent and supportive customer support experiences!

Unfortunately, since trying this out didn't work either, my management has decided that we should just use the Legacy multiplayer system until the new one is more stable (i.e. has a working UE4 dedicated server example project / tutorial).

Is there a way I could subscribe to updates about the new system (but especially about UE4 support) so that I can give it another try later this year?

0 Likes 0 ·
Show more comments
Mircea Pricop avatar image
Mircea Pricop answered

Ok I managed to get Docker running, and I discovered that my server wasn't running at all in the container.

This led me to believe this has something to do with the different OS versions, and indeed, upon RDC-ing into the running VM and attempting to run the server binary manually, I get a pop-up informing me that Visual C++ and DirectX are not provided.

I tried installing them by running the UE4PrereqSetup tool, but this also fails in the server environment (even after installing some stuff like .Net 3.5 manually).

In conclusion, the container image provided is not able to run an Unreal Engine dedicated server at the time. This unfortunately means I'll have to rely on legacy multiplayer builds only, as those VMs seem to run the server fine.

@jayzuo Do my conclusions here make sense? Is there a channel I can follow for possible future improvements to this situation? I really wanted to play nice and use the new multiplayer system (especially since the new matchmaker has some features that would make life easier).

10 |1200

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

mikecoker avatar image
mikecoker answered

When I was testing on "competing offerings", I had to create an install step to install msvcrt and directX on the machine because it wasn't included by default on a windows server. Once those were setup, my UE4 dedicated server had no issues running. I was not using docker containers though.

1 comment
10 |1200

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Mircea Pricop avatar image Mircea Pricop commented ·

That sounds promising, do you have any details about this install step and how you set it up?

0 Likes 0 ·
Lewis Mohr avatar image
Lewis Mohr answered

I was able to solve this problem by adding this line to the Build.cs file of my server module:

RuntimeDependencies.Add("$(TargetOutputDir)/xinput1_3.dll", "$(EngineDir)/Binaries/ThirdParty/AppLocalDependencies/Win64/DirectX/xinput1_3.dll", StagedFileType.SystemNonUFS);

This basically just copies the xinput dll from a folder in the engine to alongside the output binary.

The ListDlls steps above worked for me, but gave me a larger list of dlls than were actually required to run the server.

I'd also suggest when building to specify the -nobootstrapexe flag (at least if you are using RunUAT) so that it doesn't include the bootstrapper which is hardcoded to look for certain dlls and will show a message box if they don't exist, which isn't very server friendly.

1 comment
10 |1200

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Kieran Newland avatar image Kieran Newland commented ·

For anyone still struggling with this you also need to change all references of vcruntime140 in your Server.build.cs script to vcruntime140_1. It looks like the dll requirements have change and the GSDK integration guide hasn't yet been updated.

0 Likes 0 ·
Andrea avatar image
Andrea answered

Has anyone managed to get the server working with UE4? I've been trying for days and I always get the error "TooManyRestarts" and "Unhealthy".

10 |1200

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Write an Answer

Hint: Notify or tag a user in this post by typing @username.

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.