question

Benjamin Bennett avatar image
Benjamin Bennett asked

Server Instances taking a long time to clean up/recycle

Hello,

I decided to make a new post from my other topic since I learned a little bit more about the server system.

Setup:

-Unity Game Server setup with MutliplayerServerAgentAPI

-Windows

-Process Based Windows Build

-10 Servers per VM


I believe there is a bug with the server instances not cleaning up after Application.Quit() is called from a Unity Game Server that has integrated with the MultiplayerServerAgentAPI. Or at least not in a timely manner.

I confirmed the Application.Quit() is called by setting a value in my title settings right before it's reached. And on one occasion I did see a server session get recycled, but it took about 25 minutes.

But now I've hit a dead-end. I have 10 active empty sessions that are waiting to recycle and no new players can join unless I make more VMs. This doesn't seem super cost effective since the space is there.

multiplayer
playfab-0.png (31.1 KiB)
playfab-1.png (21.9 KiB)
10 |1200
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

4 Answers

·
Dimitris-Ilias Gkanatsios avatar image
Dimitris-Ilias Gkanatsios answered

can you RDP into one of these VMs and see the state of these processes? If you call Application.Quit and for whatever reason the process is alive, we will not create additional ones.

2 comments
10 |1200
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Benjamin Bennett avatar image Benjamin Bennett commented ·

Yep I did use RDP and was able to see the processes. If I kill the process in the task manager it will recycle like it's supposed to.

What's really interesting is if I make the server instances 1 per VM it works as intended. I'm doing another test with the same build for 10 instances per VM to see if breaks or works again. I'm wondering if there is an issue with my game server running as multiple processes concurrently.

0 Likes 0 ·
Benjamin Bennett avatar image Benjamin Bennett Benjamin Bennett commented ·

Can confirm that this is the issue.

If I run my game server as 1 server per VM the sessions get recycled as intended.
If I run my game server as 10 servers per VM the sessions don't get recycled and remain active indefinitely.

0 Likes 0 ·
Benjamin Bennett avatar image
Benjamin Bennett answered

After some more investigation it appears the issue is the process doesn't quit if the VM is running multiple server instances.

I tested a VM of one server per VM and the process would exit -> recycle as expected.

I tested a VM of 10 servers per VM and the process would remain around forever after Application.Quit() was called. I tried other methods of quitting too such as Environment.Exit(0) and Process.GetCurrentProcess().Kill and all the results were the same.

I can't think of any reason why this would happen as my processes aren't controlled by anything external.

My options are:

  • Figure out what is stopping the process from exiting when multiple server sessions exist. I've tried everything I can think of to debug this. I've setup error reporting to logs, setting internal title data keys and debugged a ton locally.
  • Setup my servers to handle multiple players per session and just have 1 session per VM. The downside to this is I don't believe I could use matchmaking properly and some server sessions might live a long time. I read it was a bad idea to keep around sessions for hours+.
10 |1200
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Dimitris-Ilias Gkanatsios avatar image
Dimitris-Ilias Gkanatsios answered

If you make the game server work as container, is this still an issue? I'd do a wild guess that there is a conflict between multiple instances of the same process running on the VM. Since containers isolate the process, I believe this won't create an issue.

Anything in your code that might make multiple instances of the Unity game not terminate?

8 comments
10 |1200
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Benjamin Bennett avatar image Benjamin Bennett commented ·

I'm guessing that maybe the networking threads are conflicting? I'm using the telepathy C# plugin for networking which is a basic TCP messaging system. It uses 2 threads per client (listening/sending). Maybe this is the culprit?

As for containers I did try that, but the latency is absolutely unbearable. It's like the game server is running at like 2 FPS. Not really sure why that is.

0 Likes 0 ·
Benjamin Bennett avatar image Benjamin Bennett commented ·

Yeah I just tried another container build and it's absolutely unbearable how slow it is.

0 Likes 0 ·
Dimitris-Ilias Gkanatsios avatar image Dimitris-Ilias Gkanatsios Benjamin Bennett commented ·

how are you building your container? Can you share your Dockerfile? A "container" is just a normal process with limited priviledges/access and a limited view of the filesystem

0 Likes 0 ·
Benjamin Bennett avatar image Benjamin Bennett Dimitris-Ilias Gkanatsios commented ·

I was just using the default windows container setup

0 Likes 0 ·
Benjamin Bennett avatar image Benjamin Bennett commented ·

I do see the containers cleaning up. But due to latency/speed it's unusable. I can't seem to find anything on why that would be the case.

0 Likes 0 ·
Benjamin Bennett avatar image Benjamin Bennett commented ·

@Dimitris Gkanatsios I was able to setup a Digital Ocean droplet with multiple instances of the process running and the processes were quitting as expected. I don't think the issue is on my end.

0 Likes 0 ·
Dimitris-Ilias Gkanatsios avatar image Dimitris-Ilias Gkanatsios Benjamin Bennett commented ·

do you have any special handlers for application exit/shutdown on Unity? Maybe something blocks the application termination in some way

0 Likes 0 ·
Benjamin Bennett avatar image Benjamin Bennett Dimitris-Ilias Gkanatsios commented ·

No. It just hits Application.Quit like it's supposed to. Like I said I did all the tests I could on my end and the processes were ending just fine.

0 Likes 0 ·
Benjamin Bennett avatar image
Benjamin Bennett answered

Someone from Mirror recommended I try out the KCP networking and see if that fixes the issue. So that is what I will try next.

2 comments
10 |1200
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Benjamin Bennett avatar image Benjamin Bennett commented ·

Converting it to KCP networking didn't work either.

0 Likes 0 ·
Sarah Zhang avatar image Sarah Zhang Benjamin Bennett commented ·

According to your description, you use Unity Mirror as your network framework, is it right? Could you please provide your full server sample for the reference? If you are willing to share your project, please hide all privacy information, such as developer secret key, etc. before sharing.

Besides, we tested our official Unity Mirror Server Sample and it could work fine in the process server mode. After shutting down 10 active server instances, 10 standby servers will appear quickly. Could you please try it?

0 Likes 0 ·

Write an Answer

Hint: Notify or tag a user in this post by typing @username.

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.