Playfab multiplayer server can't connect to Epic Online Services (SSL error)

Question

question

Linus Neuman asked Nov 26, '21

Playfab multiplayer server can't connect to Epic Online Services (SSL error)

Hi,

I'm currently hosting a multiplayer game server on Playfab. It's a UE4 dedicated server running on Windows. It has the playfab GSDK plugin from github integrated.

Playfab launches the server fine, and gets heartbeats.
However, it can't connect to Epic Online Services (EOS).

This is the error received:

]LogEOS: Warning: [LogHttp] Retry 5 on https://api.epicgames.dev/sdk/v1/default?platformId=WIN
[2021.11.26-13.27.59:215][458]LogEOS: Warning: [LogHttp] 0000020B0A763400: invalid HTTP response code received. URL: https://api.epicgames.dev/sdk/v1/default?platformId=WIN, HTTP code: 0, content length: 0, actual payload size: 0
[2021.11.26-13.27.59:216][458]LogEOS: Warning: [LogHttp] 0000020B0A763400: request failed, libcurl error: 60 (Peer certificate cannot be authenticated with given CA certificates)
[2021.11.26-13.27.59:216][458]LogEOS: Warning: [LogHttp] 0000020B0A763400: libcurl info message cache 10 (Hostname in DNS cache was stale, zapped)
[2021.11.26-13.27.59:216][458]LogEOS: Warning: [LogHttp] 0000020B0A763400: libcurl info message cache 11 (  Trying 34.234.116.84...)
[2021.11.26-13.27.59:217][458]LogEOS: Warning: [LogHttp] 0000020B0A763400: libcurl info message cache 12 (TCP_NODELAY set)
[2021.11.26-13.27.59:217][458]LogEOS: Warning: [LogHttp] 0000020B0A763400: libcurl info message cache 13 (Connected to api.epicgames.dev (34.234.116.84) port 443 (#22))
[2021.11.26-13.27.59:217][458]LogEOS: Warning: [LogHttp] 0000020B0A763400: libcurl info message cache 14 (ALPN, offering http/1.1)
[2021.11.26-13.27.59:217][458]LogEOS: Warning: [LogHttp] 0000020B0A763400: libcurl info message cache 15 (Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH)
[2021.11.26-13.27.59:218][458]LogEOS: Warning: [LogHttp] 0000020B0A763400: libcurl info message cache 16 (TLSv1.3 (OUT), TLS handshake, Client hello (1):)
[2021.11.26-13.27.59:218][458]LogEOS: Warning: [LogHttp] 0000020B0A763400: libcurl info message cache 17 (TLSv1.3 (IN), TLS handshake, Server hello (2):)
[2021.11.26-13.27.59:218][458]LogEOS: Warning: [LogHttp] 0000020B0A763400: libcurl info message cache 18 (TLSv1.2 (IN), TLS handshake, Certificate (11):)
[2021.11.26-13.27.59:219][458]LogEOS: Warning: [LogHttp] 0000020B0A763400: libcurl info message cache 19 (TLSv1.2 (OUT), TLS alert, Server hello (2):)
[2021.11.26-13.27.59:219][458]LogEOS: Warning: [LogHttp] 0000020B0A763400: libcurl info message cache 20 (SSL certificate problem: unable to get local issuer certificate)
[2021.11.26-13.27.59:219][458]LogEOS: Warning: [LogHttp] 0000020B0A763400: libcurl info message cache 21 (Closing connection 22)
[2021.11.26-13.27.59:220][458]LogEOS: Warning: [LogHttp] Retry exhausted on https://api.epicgames.dev/sdk/v1/default?platformId=WIN
[2021.11.26-13.27.59:220][458]LogEOS: Warning: [LogEOS] Failed to connect to the backend. ServiceName=[SDKConfig], OperationName=[GetPlatformConfigRoute], Url=[<Redacted>]
[2021.11.26-13.27.59:248][459]LogEOS: Warning: [LogEOS] SDK Config Platform Update Request Failed, Result Code: EOS_NoConnection, Retrying after 6.575274 seconds

I'm not sure why this is failing, do I need to upload a certificate to the build on playfab?
Is there a mismatch in TLS versions?

I've tried disabling bVerifyPeer in UE4, but to no avail.

This is working fine outside of Playfab, so there is some kind of environment issue I believe.

We use EOS for matchmaking, voice, etc and need our playfab server instance to be able to reach it.

Cheers,

Linus

Custom Game Servers multiplayer windows

10 |1200

Attachments: Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Answer 1 · 2022-03-23T17:15:10Z

ravarna answered Mar 23, '22

@linus neuman, the problem with the crt bundle is that windows only installs the first in the list of certificates and then stops. Amazon root is not the first one there. Here are the certs you need in the DER format from Amazon Trust Services Repository.

I think AmazonRootCA1 suffices, but you can install all of the following: RootCA1, RootCA2, RootCA3, RootCA4 and StarField.

Also, is this working on your local multiplayer agent with containerization (it may work in process mode if the root is installed on your machine).

2

10 |1200

Attachments: Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

ravarna commented · Mar 23, 2022 at 05:23 PM

Also, powershell can be tricky, I suggest having a startup.cmd file with the following contents and use that as the start game command:

certutil.exe -addstore root .\AmazonRootCA1.cer

certutil.exe -addstore root .\AmazonRootCA2.cer

certutil.exe -addstore root .\AmazonRootCA3.cer

certutil.exe -addstore root .\AmazonRootCA4.cer

certutil.exe -addstore root .\SFSRootCAG2.cer

.\GameServer.exe -log -Playfab

1 ·

Linus Neuman ravarna commented · Mar 24, 2022 at 09:17 AM

This worked beautifully, thanks!

Cheers

Linus

1 ·

Answer 2 · 2021-12-03T09:40:00Z

June Rhodes answered Dec 3, '21

@Made Wang This is not a case of the developer needing to upload custom SSL certificates, nor does EOS even have public/private key pairs.

The issue is that the container or execution environment for PlayFab servers is lacking the latest root CA certificates. Typically these are delivered through Windows Update, but for whatever reason, the PlayFab environment is missing the Amazon Root CA 1 root certificate.

Because the execution environment doesn't have the Amazon Root CA 1 certificate, the EOS SDK can't communicate with the back-end APIs.

10 |1200

Attachments: Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Answer 3 · 2021-12-14T11:44:45Z

Linus Neuman answered Dec 14, '21

Hello Made, please see my previous comment that you can't refer to June's answer since it's a call for action on your end,

Regards,
Linus

10 |1200

Attachments: Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Answer 4 · 2022-01-13T13:36:57Z

jeremy answered Jan 13, '22

@Made Wang any update on this issue ?

10 |1200

Attachments: Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Answer 5 · 2022-02-25T05:22:35Z

Nikita Matusevich answered Feb 25, '22

Is there any information on this issue?

10 |1200

Attachments: Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Answer 6 · 2022-02-25T10:30:48Z

JayZuo answered Feb 25, '22

This seems not to be the container or execution environment for PlayFab servers is lacking the latest root CA certificates.

I'm testing with the following steps:

1. Deploy a build to MPS and get a standby server.

2. Connect to the server with RDP (Here, we are actually connecting to the VM, not the server instance).

3. Open a PowerShell window as Administrator.

4. Run "docker ps" to get the Container Id of the standby server.

5. Run "docker exec -it <Container Id> powershell" to connect "inside" the running container.

6. In the running container, execute "curl.exe https://api.epicgames.dev/sdk/v1/default?platformId=WIN"

With the above command, I can get the right response. If it's the container is lacking the latest root CA certificates. I'd think the above curl command should also fail.

I'm not familiar with UE4, especially UE4 dedicated server, this issue might need further investigation.

10 |1200

Attachments: Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Answer 7 · 2022-02-28T03:35:36Z

Dimitris Gkanatsios answered Feb 28, '22

Windows containers seem to be lacking certain CAs. You can try running your game servers on process mode. This might work as the VM has more CAs integrated. To make sure your game servers work well on process mode, please try LocalMultiplayerAgent first PlayFab/MpsAgent: Azure PlayFab Multiplayer Servers LocalMultiplayerAgent project and helper libraries (github.com)

1

10 |1200

Attachments: Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Linus Neuman commented · Mar 22, 2022 at 02:45 PM

Hi!

Sorry for the late answer, we took a break from looking at cloud hosted servers.
I tested using the process method instead!
However we run into the same issue, same complaints about CA certificate in the log.
I verified it works perfectly fine in my local multiplayer agent.

0 ·

Answer 8 · 2022-03-22T06:33:59Z

admin-22 answered Mar 22, '22

Is there any development on this topic? I'm running into the same issue

10 |1200

Attachments: Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Answer 9 · 2022-03-22T22:23:52Z

Dimitris Gkanatsios answered Mar 22, '22

removed my previous answer and posting a solution we added here, let us know if it works!

MpsSamples/amazon_root_ca.md at main PlayFab/MpsSamples (github.com)

2

10 |1200

Attachments: Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Linus Neuman commented · Mar 23, 2022 at 01:26 PM

Unfortunately doesn't seem to do the trick,

If I use Windows Process mode, it fails to start instead, not sure if I did something wrong but I set something up like this:
Start command: "SetupAndLaunch.ps1".

The package zip looks like this:
GameServer.exe
SetupAndLaunch.ps1

curl-ca-bundle.crt

SetupAndLaunch.ps1 looks like this:

Import-Certificate -FilePath curl-ca-bundle.crt -CertStoreLocation 'Cert:\LocalMachine\Root' -Verbose -WhatIf

.\GameServer.exe -log -Playfab

Unfortunately, it fails to start with error message "Start servers failed". If I RDP to it, I can't see that it ever attempted to launch the game server, so perhaps it failed to run the start script or the start script failed.

If I manually launch the game via the launch script, it crashes and complains about "Heartbeat endpoint and Server id are required configuration values.", possibly because I'm launching it manually without necessary environment variables set?

If I use container mode, it doesn't seem to launch it either. It says "pending heartbeat" for a very long time.

I set the start command to "C:\Assets\SetupAndLaunch.ps1". and mounted the package zip to C:\Assets.

0 ·

Linus Neuman commented · Mar 23, 2022 at 03:29 PM

I can manually boot the game server via RDP it starts complaining about this, even if I manually install the crt bundle from curl:
[2022.03.23-15.26.25:578][658]LogEOS: Warning: [LogHttp] 000001A5C806C040: invalid HTTP response code received. URL: https://api.epicgames.dev/sdk/v1/default?platformId=WIN, HTTP code: 0, content length: 0, actual payload size: 0
[2022.03.23-15.26.25:578][658]LogEOS: Warning: [LogHttp] 000001A5C806C040: request failed, libcurl error: 60 (Peer certificate cannot be authenticated with given CA certificates)
[2022.03.23-15.26.25:578][658]LogEOS: Warning: [LogHttp] 000001A5C806C040: libcurl info message cache 23 (Hostname api.epicgames.dev was found in DNS cache)
[2022.03.23-15.26.25:578][658]LogEOS: Warning: [LogHttp] 000001A5C806C040: libcurl info message cache 24 ( Trying 35.173.6.230...)
[2022.03.23-15.26.25:579][658]LogEOS: Warning: [LogHttp] 000001A5C806C040: libcurl info message cache 25 (TCP_NODELAY set)
[2022.03.23-15.26.25:579][658]LogEOS: Warning: [LogHttp] 000001A5C806C040: libcurl info message cache 26 (Connected to api.epicgames.dev (35.173.6.230) port 443 (#5))
[2022.03.23-15.26.25:579][658]LogEOS: Warning: [LogHttp] 000001A5C806C040: libcurl info

0 ·

question