question

Juris Zebnickis avatar image
Juris Zebnickis asked

GSDK and dedicated Unreal Server not launching both processes in container

Now then!...

UE 27.2, Docker, Windows 64, Latest GSDK plugin (as of 09/05)

Hi.

We are building a server with integrated GSDK, following all the latest documentation and tutorials.

We are using Docker and following the Local Containerization guide to build and test the image and the server.

Aside from the official docs I have read all of these threads to the finest detail:

https://community.playfab.com/questions/32588/server-build-unhealthy-ue4-dedicated.html

https://community.playfab.com/questions/52890/containerized-server-not-connectable.html

https://community.playfab.com/questions/28250/mutliplayer-20-build-fails-deployment-with-noserve.html

..and there were some more. I will not go into every detail that we solved along the way, but we are now having two problems. The other 3 points are descriptive and might be relevant in the context:

1) Our server process.exe and serverProcessBinary.exe do not launch together.

So, when you launch a server locally, it will have two processes - one as the 'GameServer.exe' which is in the root folder of the build and the 'GameServerBinary.exe' which is in your binaries folder respectively. The 'shell' process has to be launched first and then the binary. This is the only way that the server will actually launch. We had the infamous 'waiting for heartbeat' due to this and after exploring the container processes with console commands we solved it by launching manually first the 'shell' and then the 'binary' part of the server with a .bat file.

This is a solution, but it is not ideal and does not explain why the server isn't launching correctly on its' own.

Screen with a local container in Active state after manual binary launch

We also tried renaming the binary so that the .exe files have the same name, as the 'shell' process has to find a binary with the same name (in theory), but this did not work either. I understand that various build configurations can make the binary.exe differ in name, but how can we account for this?

Regardless, we could not find a way to make both processes launch together as they would in a local server build. Is there something we are missing or is this a container/playfab issue? As there have been many unsuccessful attempts and unanswered posts on the forums.

2) Another issue is we do not get any log files generated even after all the above-mentioned steps. We launch both commands in our .bat file with '-log' flags, but the container-generated folder never contains a 'Saved' folder with logs as it should after it has ran. If it goes to 'Active' state and then exits after it has not received a connection after the heartbeat timeout, it should generate a log file from the Unreal run, correct? Could this be somehow related to the fact that we run the processes separately and the second one (the actual server with engine initialization part) does not get the log command for some reason?

3) The ListDll executables do not work in container mode - meaning it will never show anything for any process that is running in it. While it executes perfectly fine with any local processes. Can someone confirm/deny this on developer end?

Screen with ListDlls.exe tests

4) About additional .dll dependencies. This is just not actual anymore. Not with this generation of the Unreal engine and images. The image for the PlayFab cloud container contains all of the standard OS .dll files and has no problem getting to 'Active' state without any addition in your C++ server code.

5) Concerning the PORT numbers in the LocalMultiplayerAgent MultiplayerSettings.json. Can someone explain what exactly is the 'NodePort' and how does it differ from what the 'AgentListeningPort' does?
Because I have found in numerous posts something about changing these, but we never changed them in any of the above steps and they both still remain 56001, even though @SethDu has once mentioned that they cannot be shared. What is the potential internal error that this may cause?

Thank you.

unrealCustom Game Serversmultiplayer
10 |1200

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Dimitris-Ilias Gkanatsios avatar image
Dimitris-Ilias Gkanatsios answered

first of all, the 429 error means that you have 0 standingBy servers left in the region. Any chance you have exhausted them? Any time you do a RequestMultiplayerServer with a new session ID, you will get a new server back.
Repetitive calls of this API will result in the standingBy pool being exhausted (up to what max limit you have set on the Build).

Regarding the port connection issue, are you defining the ports correctly? Are you connecting to the IP:PORT returned by the RequestMultiplayerServer API call?

You can try using LocalMultiplayerAgent and connect from the client to the port that you have defined on the NodePort.

6 comments
10 |1200

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Juris Zebnickis avatar image Juris Zebnickis commented ·

Negatory. The set of tests that I did was done using just one unique sessionID, which was created through the MpsAllocator. Calling RequestMultiplayerServer yielded a valid active server struct, but I could not connect to it. Then at one point it became inactive as I was monitoring the PlayFab Dashboard.

Yes. I literally copypasted the values. To clarify, I allocated the server using the MpsAllocator. I did not use the matchmaking API to find it.

0 Likes 0 ·
Juris Zebnickis avatar image Juris Zebnickis commented ·

Update: Got it. I now understand that the RequestMultiplayerServer allocates a new server if the standby pool allows it. I was treating as a 'find already active multiplayer servers'. Now I got the clients to connect directly to the ip and port of the server session details manually. I tried using GetMultiplayerServerDetails method for my purposes, but this gave me back another crash, but with a different error: {"code":401,"status":"Unauthorized","error":"NotAuthorized","errorCode":1089,"errorMessage":"Only entities of the following types may call this API: title"}
1) I have a successful login callback and I can successfully call a custom cloudscript function, but why would this not be allowed for my client? There are no session tokens or anything in the input struct for it that I can find either.

2) Could you please advise what would be the best method to just find a running multiplayer server and connect to it?

0 Likes 0 ·
Juris Zebnickis avatar image Juris Zebnickis commented ·
3) Lastly, to comment on my previous attempt when I said that it's simply not connecting, this was due to having Steam running in the background on my client machine. We use steam authentication layer, but we do not use steam networking or steam net driver, however, just having steam on seems to disable access in some way. I understand that this is way out of scope, but I would be thankful for any tips.Thank you! This community is something else.
0 Likes 0 ·
Made Wang avatar image Made Wang Juris Zebnickis commented ·

1) As described by the returned error message, this API only allows the Title Entity to access.

2) You can call GetEntityToken on the server to get the Title Entity, then call GetMultiplayerServerDetails to get the IP and port, and send it to the client.

Normally, if you want to add players to a running game, you can use the server backfill ticket to match new players. If you want to let players choose and join a running game, you can try Lobby. Store the IP and port in a custom property of lobby for players to connect.

3) I don't know much about the steam network. It is recommended to go to the steam developer community for professional support.

0 Likes 0 ·
Juris Zebnickis avatar image Juris Zebnickis Made Wang commented ·

Hey, @Made Wang

I understand the part about it not being a part of client-API.

But I don't entirely understand the practical details of what you are suggesting. If I have a hosted server running, it is not itself an entity that will log into playfab. Only clients can log into playfab, no?And then even if I had that on the server, I need to have a client-side control point, like an API call that makes the server do the multiplayer server request.

Without clients connected to the server, I cannot send anything to them from the server, and I cannot connect them to the server without the server info.

Like.. am I direly misunderstanding something?

As to the steam issue, it was an FUniqueNetReplID issue that I found a solution for here:
https://community.playfab.com/questions/44075/connecting-to-playfab-server-with-steam.html

0 Likes 0 ·
Show more comments
Dimitris-Ilias Gkanatsios avatar image
Dimitris-Ilias Gkanatsios answered

1. What do you mean by "local server build"? Do you mean by using LocalMultiplayerAgent? If so, this should be using the same pattern as Azure, i.e. starting one process with StartGameCommand. Of course, this process could be a batch (as you're doing now) or PowerShell script which could launch two processes at the same time.

2. Our process orchestrator grabs whatever logs are on standard output and standard error stream of the process that is called by your StartGameCommand. Since this is Unreal, maybe you should try "-stdout" along/instead of "-log"

3-4. Happy to hear that ListDlls is not needed for your game server!

5. There is some documentation about the ports here Locally debugging game servers and integration with PlayFab - PlayFab | Microsoft Docs The NodePort is the port that your game client should communicate with. The AgentListeningPort should not be modified unless there is conflict with another process (one that already listens on 56001).

10 |1200

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Juris Zebnickis avatar image
Juris Zebnickis answered

Hi Dimitris,

Yes, we use local multiplayer agent to deploy and test the cloud environment with the playfab image provided to be as close as possible to the real thing. This does not answer my question, however, as I described above the processes should both get launched in the container, but only one of them launches with a simple .exe execution. This seems to be a widespread problem, judging by the related forum posts and requires way more in-depth digging about the process than... well. Than one would expect from a top tier hosting service.

Concerning logs, we are speaking about different things. -log is a command line flag for the server process so that all that we log in code, as well as the engine initialization logging gets output. This happens in a non-containerized launch, but does not happen within the container.

The .dll part just does not make sense. All of the reported solutions that say that a manual .dll inclusion solved their issue were also reported together with a different problem being solved, so there is no proof that it really does anything.

It still does not explain why ListDlls does not work in the container.

Yes, I have read the documentation, thank you. So, the node port, is the Network port that I name UnrealServerGsdkHostPort when uploading the build?

10 |1200

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Dimitris-Ilias Gkanatsios avatar image
Dimitris-Ilias Gkanatsios answered

First of all, sorry that you're having issues.

Concerning the launch, I'm not sure I follow 100%, sorry for that. What is your StartGameCommand? I understand that you have a batch script that launches both processes, right? We don't do anything special other than launching what you have in the StartGameCommand. We capture anything that is being sent to standard error and standard output streams and store it in the logs folder, when the process that is being executed as part of the StartGameCommand is terminated, either gracefully or via crash.

In your case, since you want to launch two executables as part of the StartGameCommand, what you can do is launch a script that i) will launch process A in the background and ii) then launch process B. The game server process should implement the GSDK so to send heartbeats to our process orchestrator so we can monitor its status.

So, my recommendation would be to use the StartGameCommand and launch a powershell script (included in your game assets) that would launch process A in the background and then launch process B normally. In this way, your "game server process lifecycle" (as known by MPS) is tied to process B lifecycle. Ideally, you should have something to monitor process A (maybe use a Powershell Register-ScheduledJob (PSScheduledJob) - PowerShell | Microsoft Docs?) so to restart it when/if it crashes.

Regarding ListDLLs, not sure if it will help at this point - let's make sure that both processes launch properly first.

The UnrealServerGsdkHostPort is the port that your game server listens to. This is the local port inside the container. On uploading the port, you don't need to know the NodePort, since it will be mapped externally to another port on the Load Balancer. Check here around minute 22 for our network architecture (14) PlayFab Multiplayer Hosted Servers part 1: Multiplayer Servers Quick Start - YouTube

Let me know if I missed anything!

10 |1200

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Juris Zebnickis avatar image
Juris Zebnickis answered

Thank you for the elaborate explanation, @Dimitris Gkanatsios!

Much appreciated for following through my possibly irrelevantly detailed scrutiny.


What I meant is that when you build any server and check the build folder afterwards, you will see a Server.exe in the root of that build folder and then a Server-Win64.exe deeper in the build folder binaries.

The first one is a small 'shell' executable, the latter one is the actual game content. This reflects in the task manager where you can look at each of the processes memory allocation and the latter will be much bigger.

So, you can launch the win64.exe binary, then launch the shell process on your own - this will show both processes in the task manager and the server will be running ok.

Or you can launch just the one in the binary folder - this will show only the binary process in the task manager.

Or you can launch the 'shell' process, which will start both processes in your build when you do this on your machine after you build it. However, this does not happen when executing in the container and only the 'shell' process shows up in the task manager, not the binary. I suspect that the issue is hidden in the naming of these executables, but I also suspect that it is an Unreal issue, not PlayFab one.

So, yes, we are launching them with a batch file and it is working alright. It's just that it took a lot of time to figure out and I hope this helps someone along the way.

A new issue appears though :)

concerning connecting to the server from an actual game client.

Everything is running correctly, it seems. We have allocation, login and multiplayer server requests, cloudscript functions working perfectly, however, when I try to bluntly just use the IP:PORT value in an 'open' command in the Unreal game client from a multiplayer server request, nothing happens.

I am currently circumventing the Matchmaking part of this process with manual allocation and copypasting the Session ID in my code, but it' s supposed to work, no? Am I missing something? Seems like all people do to connect is get IP of a running server through matchmaking and connect.

2 comments
10 |1200

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Juris Zebnickis avatar image Juris Zebnickis commented ·

Ok, so the game client started crashing with the following log after trying to connect to the server even though it is definitely called only once and the server is Active in the region:

[2022.05.13-17.47.25:214][632]LogPlayFab: Response : {"code":429,"status":"TooManyRequests","retryAfterSeconds":1,"error":"MultiplayerServerTooManyRequests","errorCode":1380,"errorMessage":"MultiplayerServerTooManyRequests - NoHostsAvailableInRegion - No Hosts available in regions 'NorthEurope', please retry."} [2022.05.13-17.47.25:215][632]LogBlueprintUserMessages: [BP_BFLoginActor_2] Cloud script error delegate: MultiplayerServerTooManyRequests - NoHostsAvailableInRegion - No Hosts available in regions 'NorthEurope', please retry.

I connected to the Virtual Machine with remote desktop and events showed nothing. I wanted to find the actual folder where the server is unpacked and running to see if there are logs there, but I couldn't find it. Is it supposed to be there somewhere?

0 Likes 0 ·
Juris Zebnickis avatar image Juris Zebnickis commented ·

I will leave the current session on, so you guys can maybe check it out.

0 Likes 0 ·
sharkraceclub avatar image
sharkraceclub answered

@juriszebnickis

Any updates on your case? I am at the same step (or close to it) as you. I have multiplayer server with Unreal Engine, I build it and run successful on LocalMultiplayerAgent, it's pretty easy with this settings in MultiplayerSettings.json :

{ "RunContainer": false,

"OutputFolder": "H:\\Game\\Packaged\\WindowsProcessLogOutput", "NumHeartBeatsForActivateResponse": 10, // default value

"NumHeartBeatsForTerminateResponse": 60, // default value

"AgentListeningPort": 56001, // default value

"TitleId": "your title ID",

"Region": "NorthEurope",

"BuildId": "your build ID",

"RunMode": 0,

"AssetDetails": [

{ "MountPath": "",

"LocalFilePath": "H:\\Game\\Packaged\\WindowsServer.zip" } ],

"PortMappingsList": [ [ { "NodePort": 30000, "GamePort": { "Name": "UnrealServerGsdkHostPort", "Number": 8888, "Protocol": "UDP" } } ] ], "ProcessStartParameters": { "StartGameCommand": "GameServer.exe -log" },

"GameCertificateDetails": [], "SessionConfig": { "SessionCookie": null, "SessionId": "593f2582-6906-4532-9da4-e0ab97cb24ec", "InitialPlayers": [] } }

It works but terminates after 60 runs and I can see log it works!

But when I deploy it to playfab clouds (I tried both process and container - last one at least work) - it shows me

I archived servers I can see some log file with no useful information. My main question is - How can I get unreal log file from cloud? I understand that my server running somehow because I seen Terminating status but I don't get why because no log file exists!

3 comments
10 |1200

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Juris Zebnickis avatar image Juris Zebnickis commented ·

Hey @sharkraceclub

We had exactly the same issue, as many others in other posts.

Seems like the documentation does not cover the details of setting execution path quite enough. Through trial and error we managed to make the containerized version work with a .bat file. I described this in detail above. This happens because your server binary process does not actually launch. Only the 'shell' process does and then as it does not receive heartbeat (because your binary process with GSDK is not running), it terminates.

0 Likes 0 ·
sharkraceclub avatar image sharkraceclub Juris Zebnickis commented ·

Thank you so much for answer, exactly problem was as you described, I had missing dlls and process just won't start. you theme is very helpful.

1 Like 1 ·
Juris Zebnickis avatar image Juris Zebnickis sharkraceclub commented ·

Good stuff. Glad it helped someone along the way.
Could you, just in case, please clarify what .dlls were missing and how did you find out they are missing?

What I am looking for is perhaps you modified something else at the same time as adding .dll dependencies, in which case might be that the .dlls were irrelevant.

0 Likes 0 ·

Write an Answer

Hint: Notify or tag a user in this post by typing @username.

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.