question

Christopher Clogg avatar image
Christopher Clogg asked

First run of game server connection issue

Hey I have been testing on Playfab (custom game servers) for the past few weeks and everything is working well except for the first time I upload a new build... or rather the first time a game instance is started on a server.

Basically, matchmaking succeeds but then the first client can't connect. However, right after this, or on any future instances of that server, there are no problems and everything runs smoothly.

I read a post on here about loading level files potentially causing a delay for first run. My level file takes roughly 100ms to load, so I made it asynchronous, but that did not help... so I profiled how long just the socket server takes to startup (on fresh servers), and it was coming back in the 5-10s range sometimes. This seems really long, especially when it takes < 1s on all future instances, and roughly 110ms on my local machine.

I am using "lidgren-network-gen3" for my server, but I had the same problem even when I was using my own custom C# socket server.

I put various delays between matchmaking and connecting into the client to alleviate this startup time issue, but even 10-12s wasn't enough. I have it at 20 seconds now which is quite a waste for this one problem.

Is there a better way way to deal with this? Is there something I am missing? I upload my builds with everything on default, set for USCentral, and my game mode is 1 to 64 players. I try waiting after the server is "Running" for a while before matchmaking, but no matter what, that first instance has this issue.

Thanks.

Custom Game Servers
10 |1200

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

brendan avatar image
brendan answered

That seems odd, as we've got quite a few people using custom game server hosting, and if this were endemic, we'd be hearing from multiple people, I'm sure. What all does your server do on start, specifically? Can you try creating a log file which records all activity from exe launch through to the first connections, to see if you can find any indication of why there might be delays? If the executable has a thread listening for a connection on the correct port from when it is launched, I would not expect to see this.

10 |1200

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Christopher Clogg avatar image
Christopher Clogg answered

Thanks for the quick response! Something must be going on so I am hopeful to find it haha.

My server code is reminiscent of the C# example project. I also tried removing my async level loading code completely just to test bare-bones, and added C# stopwatch profiling in my main method.

The code consists of parsing the command line args, then starting the lidgren-network server (just like starting a socket server).

With no level-loading code:
Socket server took 1.1s from the start of main method to being ready. I removed this build and re-uploaded to test a second time, and it took 0.84s.
(both of these cases were the first instance to be run on their server).

After both tests I also profiled subsequent instances and the average was around ~0.15s. So while 1.1s & 0.84s were much better than some of my previous tests, the first run on a server still takes almost 10x longer than future instances.

Added my level-loading code back in:
First test came in at 3.5s. So it appears it's having an effect even when async. However I did a 2nd fresh build test and it came in at 0.9s... so quite variable overall and almost the same as my original tests. Perhaps if I kept testing I would hit some average that isn't much more than without level files. Also, just as in my original post, all future instances took ~0.15s.

So finally I figure... why not try some fresh & simple code? I put together a very simple C# TCP server that starts up and then ends:

static void Main(string[] args)
        {
            var watch = System.Diagnostics.Stopwatch.StartNew();

            foreach (var arg in args)
            {
                var argArray = arg.Split('=');
                if (argArray.Length < 2)
                {
                    continue;
                }
                var key = argArray[0].Contains("-") ? argArray[0].Replace("-", "").Trim() : argArray[0].Trim();
                var value = argArray[1].Trim();


                switch (key.ToLower())
                {
                    case "game_id":
                        ulong gameId;
                        ulong.TryParse(value, out gameId);
                        GameId = gameId;
                        break;
                    case "game_build_version":
                        GameBuildVersion = value;
                        break;
                    case "game_mode":
                        GameMode = value;
                        break;
                    case "server_host_domain":
                        ServerHostDomain = value;
                        break;
                    case "server_host_port":
                        var hostPort = 0;//this is unity networkings default port.
                        int.TryParse(value, out hostPort);
                        hostPort = hostPort > 0 ? hostPort : 7777;
                        ServerHostPort = hostPort;
                        break;
                    case "server_host_region":
                        ServerHostRegion = value;
                        break;
                    case "playfab_api_endpoint":
                        PlayFabApiEndpoint = value;
                        break;
                    case "title_secret_key":
                        TitleSecretKey = value;
                        break;
                    case "custom_data":
                        CustomData = value;
                        break;
                    case "log_file_path":
                        LogFilePath = value;
                        break;
                    case "output_files_directory_path":
                        OutputFilesDirectory = value;
                        break;
                    case "title_id":
                        TitleId = value;
                        break;
                }


                if (!string.IsNullOrEmpty(TitleSecretKey))
                {
                    PlayFabSettings.DeveloperSecretKey = TitleSecretKey;
                }


                if (!string.IsNullOrEmpty(TitleId))
                {
                    PlayFabSettings.TitleId = TitleId;
                }
            }


            if (GameId >= 0)
            {
                LogEvent(string.Format("GameId:{0} game_build_version:{1} server:{2} port:{3} endoint:{4}", GameId, GameBuildVersion, ServerHostDomain, ServerHostPort, PlayFabApiEndpoint));
            }


            LogEvent("Setting up server...");
            /* Initializes the Listener */
            IPEndPoint localEndPoint = new IPEndPoint(IPAddress.Any, ServerHostPort);
            TcpListener myList = new TcpListener(localEndPoint);


            /* Start Listeneting at the specified port */
            myList.Start();


            myList.BeginAcceptTcpClient(HandleClientConnection, myList);


            LogEvent("Server setup complete");


            watch.Stop();
            var elapsedMs = watch.ElapsedMilliseconds;
            LogEvent("server took " + (elapsedMs / 1000.0).ToString() + "s");


            /* clean up */
            myList.Stop();


            SaveLogs();


            Thread.Sleep(2000);

	    // program over
        }


        private static void HandleClientConnection(IAsyncResult result)
        {
            var server = result.AsyncState as TcpListener;
        }


        private static void LogEvent(string logEvent)
        {
            logEvent = DateTime.Now.ToString() + ": " + logEvent;
            Console.WriteLine(logEvent);
            if (!isLocalServer)
                LogEntries.Add(logEvent);
        }


        private static void SaveLogs()
        {
            if (!string.IsNullOrEmpty(LogFilePath))
            {
                if (LogEntries.Count > 0)
                {
                    if (!File.Exists(LogFilePath))
                    {
                        LogEntries.Add(Environment.NewLine);
                        File.WriteAllLines(LogFilePath, LogEntries.ToArray());
                    }
                    else
                    {
                        File.AppendAllLines(LogFilePath, LogEntries.ToArray());
                    }
                    LogEntries.Clear();
                }
            }
        }

The result is that on first instance, it took 0.389s. This number seems low until I ran 2nd+ instances and they took ~0.009s. (My local machine took around 0.005s every time). So while the Lidgren-Network socket server is slower to startup than a TCP Listener, it's evident that no matter what the code, there is some weird slowness for the first instance on a server.

I wonder if it is Windows loading the frameworks or something? Or do the Amazon servers start in a sleep mode? I can change my code up, but if the number is wildly inconsistent then it's hard to pick some arbitrary delay for all clients (not to mention pointless if the rest of game instances take approx 0.1s). Unless I just build in some X second delay and a secret reconnect on the client if it fails within the 2 mins of a matchmaking ticket.

Any insight is appreciated, thanks!

2 comments
10 |1200

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

brendan avatar image brendan commented ·

The servers don't start in a sleep mode, so I suspect it's the frameworks. Now that said, you won't get this down to zero difference between the first instance start time on a server and subsequent starts. The server is running, and the game instance logic has been set up in the correct folder, but the time to load it will always be slower the first time than subsequent instances. In most cases, I would expect the latency time to the service from the player's location to get you past that initial start time, but it's a good idea to have a retry on the call to the dedicated server, just in case.

0 Likes 0 ·
Christopher Clogg avatar image Christopher Clogg brendan commented ·

Ok will do, thanks!

0 Likes 0 ·
Nicholas Maxey avatar image
Nicholas Maxey answered

Did you ever find a solution to this? We are having the same issue with the first game instance on a new build

1 comment
10 |1200

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Nicholas Maxey avatar image Nicholas Maxey commented ·
0 Likes 0 ·

Write an Answer

Hint: Notify or tag a user in this post by typing @username.

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.