artifact: permission error to read certificates

[ahjohannessen]

I got this on Flatcar Linux this morning:

failed to download artifact "https://github.com/grpc-ecosystem/grpc-health-probe/releases/download/v0.4.35/grpc_health_probe-linux-amd64": getter subprocess failed: exit status 1: failed to download artifact: Get "https://github.com/grpc-ecosystem/grpc-health-probe/releases/download/v0.4.35/grpc_health_probe-linux-amd64": tls: failed to verify certificate: x509: failed to load system roots and no roots provided; open /etc/ssl/certs/ca-certificates.crt: permission denied

Seems something changed with regards to artifact permissions to read certificates:

tls: failed to verify certificate: x509: failed to load system roots and no roots provided; open /etc/ssl/certs/ca-certificates.crt: permission denied

After upgrading to 1.9.1 -> 1.9.3. Temporarily solved it by setting disable_filesystem_isolation = true, which probably is not a permanent fix or good idea?

On Fedora CoreOS machines I do not have this issue (yet).

Nomad version

1.9.3

Operating system and Environment details

Flatcar Container Linux

Flatcar Container Linux by Kinvolk stable 4081.2.0 for VMware
core@app03 ~ $ uname -a
Linux app03 6.6.60-flatcar #1 SMP PREEMPT_DYNAMIC Tue Nov 12 16:20:46 -00 2024 x86_64 Intel(R) Xeon(R) Gold 6138 CPU @ 2.00GHz GenuineIntel GNU/Linux

Fedora CoreOS:

Fedora CoreOS 41.20241027.3.0
core@app04:~$ uname -a
Linux app04 6.11.5-300.fc41.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Oct 22 20:11:15 UTC 2024 x86_64 GNU/Linux

[tgross]

On Fedora CoreOS machines I do not have this issue (yet).

@ahjohannessen when you say you don't have this issue on CoreOS but you do on Flatcar, are you talking about the exact same version of Nomad? Also, don't both those distros run all the software as containers?

[ahjohannessen]

On Fedora CoreOS machines I do not have this issue (yet).

@ahjohannessen when you say you don't have this issue on CoreOS but you do on Flatcar, are you talking about the exact same version of Nomad? Also, don't both those distros run all the software as containers?

@tgross

Same version of Nomad. I install the binaries with ansible-nomad, no container install.

For things like consul, consul-template, nomad and vault I prefer setting it up running outside containers. Everything else goes into containers that Nomad controls :)

[tgross]

Very puzzling... our Landlock library didn't change between 1.9.1 and 1.9.3 (we just upgraded it but that's not in shipped versions yet). #24157 landed in 1.9.2 but I don't see any way in which that could impact permissions for the getter subprocess, because (a) it only kicks in if you ask for it, and (b) it's applied after the artifact is downloaded, which is later than you see here. The go-getter library was updated for 1.9.0, so any change there would have impacted your 1.9.1 deployment as well.

A few more things for us to look at:

• Can you verify the file permissions are identical between the two hosts?
• Can you post the kernel version for both hosts, and check whether landlock is enabled for either?
• Can you post the full log-line with a few lines of before-and-after context?