question

robertdemjan avatar image
robertdemjan asked

Custom game server logs

Hello!

I've been fighting with a problem that causes the game instances not shutting down properly. I've put many logs in the server executable, so I could check at which point the server refuses the shutting down procedure.

However, it seems like playfab is clearing out old log files? One session is played for average 10 minutes, could extend to 1 hour maybe. So whenever I see a game with longer than 1 hour uptime ( and usually no one in the lobby anymore ), I terminate them.

However, I cannot gather the logs from them. It doesn't show up in the "Archived Games" tab, or at least not at the first position. It seems like that list is ordered by starting time, not ending time. Considering that we have actively around 50 games running at the same time, that means we have 50 games opening up at every 10 minutes. That means in 1 hour, I have 300 sessions. At the time I terminate this stuck session, and the game gets in to the archived games list, it will be at the 300th place. Considering it's loading 20 items per page, that means I have to click 15 times the "Load more" button. At the time I could get to the right game, it will be flooded again.

But this is not the worst. Even if I manage to find it, when I click on Log, or output files, it will just show me blank page. Seems like "too old" logs are getting removed. None of the logs around the stuck instance are recoverable.

This is kind of problematic, because it's hard to find the error without proper logs.


An example:

Title ID: 2ABB

EntityID: 708961B8D64C2601

According to how the other sessions log URL generated, it should be here:

https://developer.playfab.com/en-us/2ABB/servers/archived-games/708961B8D64C2601/log

Is there anything I could do to get these logs?

Thanks for the help

Custom Game Servers
10 |1200
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

1 Answer

·
brendan avatar image
brendan answered

Actually, no, we don't remove older log files. Looking at your title, we have log files dating back to when you first started testing server builds, in February of 2017.

If the log file isn't there, it's going to be for one of the following reasons:

1. The log file was not written, or was written to the wrong location/filename. Sounds like that's not the case here, since you do have other log files.

2. The log file was too large, and so couldn't be copied from the server in the time available before shutdown (100 MB log files definitely won't be successfully copied).

3. (assuming you're using our current hosting service) The EC2 server host machine crashed (should be extremely rare, but it can happen).

But looking at the log storage for your title, I can confirm that we have no log for Lobby ID 708961B8D64C2601, so one of those three reasons would be the cause.

13 comments
10 |1200
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

robertdemjan avatar image robertdemjan commented ·

That's really strange. I seriously doubt that the log file would be larger that 100MB, since other files are around 60kB. We don't really log anything during the game, only when there are some events.

First case shouldn't be, usually it works.

We are using your hosting service, yes. Does it copy the log file with the output files? Because both is empty. Does the file size limit restricts the log file and the output together?

I don't think that server host crashed, there were other games running just fine on that computer.

Although it is possible that our log is getting too big because something bad is happening and it's logging the problem every time. Do you have any error logging when the copy fails? Can we somehow make sure that this is the case, and the files are getting too large? That would be a start what I could check, and maybe prevent it in application.

0 Likes 0 ·
brendan avatar image brendan robertdemjan commented ·

The logic is that when the executable which was launched by our hosting (the one specified in the build) exits, we copy the log file (log_file_path) and any output files to S3. To date, the only failures to copy those files are the ones listed above, but if the output files are also many MB, that could be a consideration - we haven't tested that exhaustively, but it does seem likely, since the time required to copy the data doesn't change between the two folders. What all are you attempting to save in the output files?

Unfortunately, we don't have specific logging for a copy operation that was stopped due to excessive file size - the ability to do more direct debugging of servers is actually one of the main reasons we're moving to our new V2 hosting model (https://blog.playfab.com/blog/introducing-playfab-multiplayer-servers). We'll be making that public as soon as we can.

0 Likes 0 ·
robertdemjan avatar image robertdemjan brendan commented ·

Shouldn't be anything big in the output folder. What I know of, there should be some other, smaller log files. Why I asked is because these servers missing output files too. if the file limit would be different for the log file and the output, i should be able to see one of them.

0 Likes 0 ·
Show more comments
robertdemjan avatar image robertdemjan commented ·

At the moment, no. I've been manually terminating these instances. I'll post the ID as soon as I find one. Thank you!

0 Likes 0 ·
robertdemjan avatar image robertdemjan robertdemjan commented ·

@Brendan I have one server currently running for 1 hour, seemingly without players.
ServerId: i-0823940f01bff73ec
LobbyID: C92052C53350E989

0 Likes 0 ·
robertdemjan avatar image robertdemjan robertdemjan commented ·

Seems to be closed down now, another one:

serverID i-091a94e8576c59e28

LobbyID 5ACA09CEF01973AD

0 Likes 0 ·
brendan avatar image brendan robertdemjan commented ·

Looks like it's not running, now. With today being a national holiday, it's a bit difficult to coordinate on times. We'll have more folks in the office tomorrow, so if you post one then, we can have a look.

0 Likes 0 ·
Show more comments

Write an Answer

Hint: Notify or tag a user in this post by typing @username.

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.