Nomad 1.7.6 does not pick up Docker DNS settings #20174

Jess3Jane · 2024-03-20T22:29:05Z

Nomad version

Version in which this functionality is broken:

root@client-1:/# nomad version
Nomad v1.7.6
BuildDate 2024-03-12T07:27:36Z
Revision 594fedbfbc4f0e532b65e8a69b28ff9403eb822e

Version in which this functionality is working (I tested down to 1.7.2 and they all behave the same):

root@client-1:/# nomad version
Nomad v1.7.5
BuildDate 2024-02-13T15:10:13Z
Revision 5f5d4646198d09b8f4f6cb90fb5d50b53fa328b8

Operating system and Environment details

root@client-1:/# uname -a
Linux client-1 5.15.0-67-generic #74-Ubuntu SMP Wed Feb 22 14:14:39 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
root@client-1:/# lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 22.04.2 LTS
Release:	22.04
Codename:	jammy

This is a completely fresh digital ocean node (I will note all of the changes I made below). Of note we have also hit this issue on systems of varying other configurations (all Ubuntu 22.04 x64).

Issue

Docker allows you to configure a set of DNS servers to give to each container. In Nomad 1.7.5 if you did not set the DNS settings for a job using the docker driver the containers in that job would have this DNS configuration. In Nomad 1.7.6 the containers instead have the host's default DNS configuration which is often undesirable.

Reproduction steps

Spin up a fresh Ubuntu 22.04 server (I used a Digital Ocean droplet for our reproduction but we've noticed this happening across our fleet so I don't think they're doing anything weird).
Install docker-ce as per their docs (I used Docker's apt registry to install it).
Install Nomad as per the docs (for the reproduction I specifically used the version of Nomad from Hashicorps repos).
Install the base CNI plugins by placing the contents of https://github.com/containernetworking/plugins/releases/download/v1.0.0/cni-plugins-linux-amd64-v1.0.0.tgz into /opt/cni/bin
Use a dropin file to configure DNS servers for docker. Specifically, I placed the following file at /etc/systemd/system/docker.service.d/overrides.conf:

[Service]
ExecStart=
ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock --dns=8.8.8.8

systemctl daemon-reload
systemctl start docker
systemctl start nomad
nomad job run <job-file.hcl>

Expected Result

If we exec into our task on Nomad 1.7.5 we can observe the correct value of resolv.conf

root@client-1:/# nomad alloc exec -i -t -task test d4de6479 cat /etc/resolv.conf
# Generated by Docker Engine.
# This file can be edited; Docker Engine will not make further changes once it
# has been modified.

nameserver 8.8.8.8
search .

# Based on host file: '/run/systemd/resolve/resolv.conf' (legacy)
# Overrides: [nameservers]

Compare this to the result of running a container with docker directly to see that they match:

root@client-1:/# docker run -it busybox cat /etc/resolv.conf
# Generated by Docker Engine.
# This file can be edited; Docker Engine will not make further changes once it
# has been modified.

nameserver 8.8.8.8
search .

# Based on host file: '/run/systemd/resolve/resolv.conf' (legacy)
# Overrides: [nameservers]

Actual Result

If you do exactly the same with Nomad 1.7.6 you instead will find that the job has a different resolv.conf:

root@client-1:/etc/systemd/system# nomad alloc exec -i -t -task test f39dae59 cat /etc/resolv.conf
# This is /run/systemd/resolve/resolv.conf managed by man:systemd-resolved(8).
# Do not edit.
#
# This file might be symlinked as /etc/resolv.conf. If you're looking at
# /etc/resolv.conf and seeing this text, you have followed the symlink.
#
# This is a dynamic resolv.conf file for connecting local clients directly to
# all known uplink DNS servers. This file lists all configured search domains.
#
# Third party programs should typically not access this file directly, but only
# through the symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a
# different way, replace this symlink by a static file or a different symlink.
#
# See man:systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.

nameserver 67.207.67.2
nameserver 67.207.67.3
nameserver 67.207.67.2
# Too many DNS servers configured, the following entries may be ignored.
nameserver 67.207.67.3
nameserver 67.207.67.2
nameserver 67.207.67.3
search .

The exact values will likely differ on your system but we can confirm that this is the contents of /run/systemd/resolv/resolv.conf:

root@client-1:/# cat /run/systemd/resolve/resolv.conf
# This is /run/systemd/resolve/resolv.conf managed by man:systemd-resolved(8).
# Do not edit.
#
# This file might be symlinked as /etc/resolv.conf. If you're looking at
# /etc/resolv.conf and seeing this text, you have followed the symlink.
#
# This is a dynamic resolv.conf file for connecting local clients directly to
# all known uplink DNS servers. This file lists all configured search domains.
#
# Third party programs should typically not access this file directly, but only
# through the symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a
# different way, replace this symlink by a static file or a different symlink.
#
# See man:systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.

nameserver 67.207.67.2
nameserver 67.207.67.3
nameserver 67.207.67.2
# Too many DNS servers configured, the following entries may be ignored.
nameserver 67.207.67.3
nameserver 67.207.67.2
nameserver 67.207.67.3
search .

Job file (if appropriate)

This happens with all jobs that don't configure any dns settings but the specific job I've used for testing is:

job "jess-test-job" {
	type = "service"
	datacenters = ["*"]
	group "test" {
		network {
			mode = "bridge"
		}
		task "test" {
			driver = "docker"
			config {
				args = ["sleep", "infinity"]
				image = "busybox"
			}
		}
	}
}

The text was updated successfully, but these errors were encountered:

apollo13 · 2024-03-21T07:17:04Z

@tgross This awfully feels like 45b2c34 ?

lgfa29 · 2024-03-21T22:22:39Z

Thank you for the detailed report @Jess3Jane and thank you @apollo13 for the git log spelunking. I was able to verify that reverting that commit does fix the problem.

I suspect we need to guard the DNS config override to explicit cni/ networks, so the bridge network is not affected, but I'm not sure if this would also revert the intended fix.

nomad/client/allocrunner/taskrunner/task_runner.go

Lines 1137 to 1145 in 23e4b7c

    
           // set DNS from any CNI plugins 
        
           netStatus := tr.allocHookResources.GetAllocNetworkStatus() 
        
           if netStatus != nil && netStatus.DNS != nil { 
        
           	dns = &drivers.DNSConfig{ 
        
           		Servers:  netStatus.DNS.Servers, 
        
           		Searches: netStatus.DNS.Searches, 
        
           		Options:  netStatus.DNS.Options, 
        
           	} 
        
           }

I've placed the issue for further roadmapping.

apollo13 · 2024-03-22T06:28:18Z

Yeah I do not think we can solely do this for CNI/ networks. This is needed for the default bridge with transparent proxy as well because then the consul-k8s plugin will provide a DNS server.

…

On Thu, Mar 21, 2024, at 23:23, Luiz Aoqui wrote: Thank you for the report @Jess3Jane <https://github.com/Jess3Jane> and thank you @apollo13 <https://github.com/apollo13> for the git log spelunking. I was able to verify that reverting that commit does fix the problem. I suspect we need to guard the DNS config override to explicit `cni/` networks, so the `bridge` network is not affected, but I'm not sure if this would also revert the intended fix. https://github.com/hashicorp/nomad/blob/23e4b7c9d23350f9d3bd2707b0d79f413767c438/client/allocrunner/taskrunner/task_runner.go#L1137-L1145 I've placed the issue for further roadmapping. — Reply to this email directly, view it on GitHub <#20174 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAT5C5JDED5BUSM45NMUALYZNMUNAVCNFSM6AAAAABFALGUB2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMJTHE2TIMJVGE>. You are receiving this because you were mentioned.Message ID: ***@***.***>

tgross · 2024-03-22T13:18:37Z

Hi folks! This is certainly a regression caused by the bug fix I did in #20007. I had a comment in that PR:

there are three potential sources for DNS configuration here: the output of CNI plugins, the user-provided DNS config, and whatever is on the client

It turns out there are four! At first glance prioritizing the four different sources would be challenging because of the work I'm working on for transparent proxy (as @apollo13 noted). But a quick check of spew.Dump shows that the DNS value we get back from mode="bridge" isn't nil, but it is empty. ~~So that's likely a simple bug at task_runner.go#L1131 where that value is never nil, and we should be checking whether it actually has any values.~~

I'll do some testing of this and likely get a PR up later today if that proves to be the correct hypothesis. Thanks again @Jess3Jane, @apollo13, and @lgfa29!

Edit: bah that's two different structs, as we convert from a non-pointer CNI DNS result to a Nomad-internal struct pointer, so that's not the problem exactly but I'm sure it's something along those lines. Still investigating.

tgross · 2024-03-22T13:44:46Z

Ok, I've got it. The cni.Result returns a slice of DNS entries, and if there's no DNS it returns a single DNS struct that is empty (because it's not a pointer). So there's a bug in the original code in networking_cni.go around how we handle the fallback in the no-DNS case, but that bug was harmless until #20007 because we always through the DNS value away anyways 🤦

Fix should be easy, just working up some unit tests to make sure the behavior is properly exercised as well.

In #20007 we fixed a bug where the DNS configuration set by CNI plugins was not threaded through to the task configuration. This resulted in a regression where a DNS override set by `dockerd` was not respected for `bridge` mode networking. Our existing handling of CNI DNS incorrectly assumed that the DNS field would be empty, when in fact it contains a single empty DNS struct. Handle this case correctly by checking whether the DNS struct we get back from CNI has any nameservers, and ignore it if it doesn't. Expand test coverage of this case. Fixes: #20174

tgross · 2024-03-22T14:23:06Z

Fixed in #20189

In #20007 we fixed a bug where the DNS configuration set by CNI plugins was not threaded through to the task configuration. This resulted in a regression where a DNS override set by `dockerd` was not respected for `bridge` mode networking. Our existing handling of CNI DNS incorrectly assumed that the DNS field would be empty, when in fact it contains a single empty DNS struct. Handle this case correctly by checking whether the DNS struct we get back from CNI has any nameservers, and ignore it if it doesn't. Expand test coverage of this case. Fixes: #20174

tgross · 2024-03-22T14:59:05Z

This fix will get shipped in the next release of Nomad, as well as backported to 1.6.x and 1.5.x

…rd into release/1.6.x (#20192) In #20007 we fixed a bug where the DNS configuration set by CNI plugins was not threaded through to the task configuration. This resulted in a regression where a DNS override set by `dockerd` was not respected for `bridge` mode networking. Our existing handling of CNI DNS incorrectly assumed that the DNS field would be empty, when in fact it contains a single empty DNS struct. Handle this case correctly by checking whether the DNS struct we get back from CNI has any nameservers, and ignore it if it doesn't. Expand test coverage of this case. Fixes: #20174 Co-authored-by: Tim Gross <tgross@hashicorp.com>

…rd into release/1.5.x (#20191) In #20007 we fixed a bug where the DNS configuration set by CNI plugins was not threaded through to the task configuration. This resulted in a regression where a DNS override set by `dockerd` was not respected for `bridge` mode networking. Our existing handling of CNI DNS incorrectly assumed that the DNS field would be empty, when in fact it contains a single empty DNS struct. Handle this case correctly by checking whether the DNS struct we get back from CNI has any nameservers, and ignore it if it doesn't. Expand test coverage of this case. Fixes: #20174 Co-authored-by: Tim Gross <tgross@hashicorp.com>

In #20007 we fixed a bug where the DNS configuration set by CNI plugins was not threaded through to the task configuration. This resulted in a regression where a DNS override set by `dockerd` was not respected for `bridge` mode networking. Our existing handling of CNI DNS incorrectly assumed that the DNS field would be empty, when in fact it contains a single empty DNS struct. Handle this case correctly by checking whether the DNS struct we get back from CNI has any nameservers, and ignore it if it doesn't. Expand test coverage of this case. Fixes: #20174

Jess3Jane added the type/bug label Mar 20, 2024

lgfa29 added theme/networking theme/cni stage/accepted Confirmed, and intend to work on. No timeline committment though. labels Mar 21, 2024

tgross self-assigned this Mar 22, 2024

tgross added this to the 1.7.x milestone Mar 22, 2024

tgross mentioned this issue Mar 22, 2024

cni: fix regression in falling back to DNS owned by dockerd #20189

Merged

tgross closed this as completed in #20189 Mar 22, 2024

tgross added this to Nomad - Community Issues Triage Jun 24, 2024

tgross moved this to Done in Nomad - Community Issues Triage Jun 24, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Nomad 1.7.6 does not pick up Docker DNS settings #20174

Nomad 1.7.6 does not pick up Docker DNS settings #20174

Jess3Jane commented Mar 20, 2024

apollo13 commented Mar 21, 2024

lgfa29 commented Mar 21, 2024 •

edited

Loading

apollo13 commented Mar 22, 2024 via email

tgross commented Mar 22, 2024 •

edited

Loading

tgross commented Mar 22, 2024

tgross commented Mar 22, 2024

tgross commented Mar 22, 2024

Nomad 1.7.6 does not pick up Docker DNS settings #20174

Nomad 1.7.6 does not pick up Docker DNS settings #20174

Comments

Jess3Jane commented Mar 20, 2024

Nomad version

Operating system and Environment details

Issue

Reproduction steps

Expected Result

Actual Result

Job file (if appropriate)

apollo13 commented Mar 21, 2024

lgfa29 commented Mar 21, 2024 • edited Loading

apollo13 commented Mar 22, 2024 via email

tgross commented Mar 22, 2024 • edited Loading

tgross commented Mar 22, 2024

tgross commented Mar 22, 2024

tgross commented Mar 22, 2024

lgfa29 commented Mar 21, 2024 •

edited

Loading

tgross commented Mar 22, 2024 •

edited

Loading