-
Notifications
You must be signed in to change notification settings - Fork 21
Troubleshooting Guide
Under normal circumstances and if there are no conflicts with other software, you should not experience any issues. That being said, nothing is perfect so if you encounter any issues with the ctrld utility (it stops resolving DNS queries), here are some useful things to watch out for, and provide to support when opening an issue.
Below assumes you're using a Linux-like operating system.
First, let's check to see if ctrld is still running.
ps aux | grep ctrld
Alternatively, if you get an error from the above command, especially on routers you may need to run:
ps | grep ctrld
or:
ps w | grep ctrld
If you don't see a running ctrld process, that would be an issue. Try restarting using the ctrld start command.
if the process is running, let's dig deeper. ctrld will spawn one or more DNS listeners in order to receive DNS queries. Let's check what they are:
netstat -tupln | grep ctrld
On MacOS you need to run this instead:
sudo lsof -iUDP -nP | grep ctrld
You may need to prefix the above command with sudo if you're not using a root user. The output may look something like this
root@UDMPro:/tmp# netstat -tupln | grep ctrld
tcp6 0 0 :::5354 :::* LISTEN 287225/ctrld
udp 0 0 0.0.0.0:5353 0.0.0.0:* 287225/ctrld
udp6 0 0 :::5354 :::* 287225/ctrld
You can ignore the listeners running on UDP 5353, as that's used for mDNS client discovery. The other instances on port 5354 (yours may be different) are the DNS listeners we're interested in.
If you don't see any listeners, or nothing except 5353, that suggests you may have a configuration problem as no DNS listener was spawned. This is very unlikely, as the service wouldn't have started in the first place. You should check your ctrld.toml config file, it's typically located in /etc/controld/ directory but can be different on some routers (Asus Merlin, DD-WRT, Tomato) and see what listener port is defined.
If ctrld is not listening on port 53, that means it's an upstream for another DNS server, usually dnsmasq. To rule out ctrld as the cause of the problem, let's make a DNS query directly against it, bypassing dnsmasq.
If you have dig installed, run:
dig verify.controld.com @127.0.0.1 -p5354
If all is well, you should get the following response
root@UDMPro:/tmp# dig verify.controld.com @127.0.0.1 -p5354
; <<>> DiG 9.16.42-Debian <<>> verify.controld.com @127.0.0.1 -p5354
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 40286
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;verify.controld.com. IN A
;; ANSWER SECTION:
verify.controld.com. 20 IN CNAME api.controld.com.
api.controld.com. 24 IN A 147.185.34.1
;; Query time: 0 msec
;; SERVER: 127.0.0.1#5354(127.0.0.1)
;; WHEN: Fri Sep 08 19:36:41 EDT 2023
;; MSG SIZE rcvd: 71
If you don't have dig installed, you can try using nslookup
root@UDMPro:~# nslookup -port=5354 verify.controld.com 127.0.0.1
Server: 127.0.0.1
Address: 127.0.0.1#5354
Non-authoritative answer:
verify.controld.com canonical name = api.controld.com.
Name: api.controld.com
Address: 147.185.34.1
Name: api.controld.com
Address: 2606:1a40:3::1
If you're getting the above responses, then ctrld is working correctly, and the issue lies elsewhere (read on!). If you do not get the above response while using a Control D DNS resolver, then something is wrong here, which can include:
- Lack of Internet connectivity
- Service encountered some unexpected state
If you have Internet access (can ping external IPs), then you should restart the ctrld service using the ctrld restart command and see if it comes back to life and DNS starts working.
If the above step was successful, yet DNS is still broken, you should check your /etc/resolv.conf file. Under normal circumstances, it will look like this:
root@UDMPro:~# cat /etc/resolv.conf
# resolv.conf(5) file generated by ctrld
# DO NOT EDIT THIS FILE BY HAND -- CHANGES WILL BE OVERWRITTEN
nameserver 127.0.0.1
root@UDMPro:~#
If your resolv.conf file has no mentions of ctrld, that means another process rendered this file. This can be other DNS related software, so you should uninstall whatever you have running that may be trying to do the same things as ctrld. If it looks like the above, let's continue.
The above assumes 127.0.0.1:53, which means your DNS queries are being sent to whichever process is listening on this IP + port. In most cases it will be dnsmasq, unless your ctrld instances runs on port 53 directly, but since we already tested it in the earlier step this is likely not the case for you. Let's confirm this.
root@UDMPro:~# netstat -tupln | grep ":53\b"
tcp 0 0 127.0.0.1:53 0.0.0.0:* LISTEN 1315695/dnsmasq
tcp 0 0 192.168.1.1:53 0.0.0.0:* LISTEN 1315695/dnsmasq
tcp 0 0 127.0.2.1:53 0.0.0.0:* LISTEN 1/init
tcp6 0 0 ::1:53 :::* LISTEN 1315695/dnsmasq
tcp6 0 0 fe80::6022:32ff:fea1:53 :::* LISTEN 1315695/dnsmasq
udp 0 0 127.0.0.1:53 0.0.0.0:* 1315695/dnsmasq
udp 0 0 192.168.1.1:53 0.0.0.0:* 1315695/dnsmasq
udp 0 0 127.0.2.1:53 0.0.0.0:* 1/init
udp6 0 0 ::1:53 :::* 1315695/dnsmasq
udp6 0 0 fe80::6022:32ff:fea1:53 :::* 1315695/dnsmasq
root@UDMPro:~#
The above indeed suggests dnsmasq is running on port 53, on all IPs. If your earlier dig/nslookup commands worked against the IP + port that ctrld listens on, but do NOT work against port 53, the issues lies in dnsmasq, and it not sending DNS queries to ctrld.
The cause for this can be other DNS related software manipulating dnsmasq config file, misconfiguration on your part, or a ctrld bug. Yeah, this is not very helpful, but you should find your dnsmasq config file and see if the server param is set correctly, pointing to your ctrld listener IP + port. If all is well, it will look something like this, but your path may be different.
root@UDMPro:~# cat /run/dnsmasq.conf.d/zzzctrld.conf
# GENERATED BY ctrld - DO NOT MODIFY
no-resolv
server=127.0.0.1#5354
add-mac
root@UDMPro:~#
Other paths to look in, depending on your hardware.
/etc/dhcpd/dhcpd-zzz-ctrld.conf
/etc/dnsmasq.d/dnsmasq-zzz-ctrld.conf
/tmp/dnsmasq.d/ctrld.conf
/home/pi/.firewalla/config/dnsmasq_local/ctrld
If this config doesn't exist, or has a different IP + port, that's a problem. Try restarting the ctrld service and see if the issue goes away.
On Merlin routers, you may want to check if /jffs/scripts/dnsmasq.postconf exists and contains following content:
root@RT-AX56U-4F98:/jffs/controld# cat /jffs/scripts/dnsmasq.postconf
# GENERATED BY ctrld - DO NOT MODIFY
#!/bin/sh
config_file="$1"
. /usr/sbin/helper.sh
pid=$(cat /tmp/ctrld.pid 2>/dev/null)
if [ -n "$pid" ] && [ -f "/proc/${pid}/cmdline" ]; then
pc_delete "servers-file" "$config_file" # no WAN DNS settings
pc_append "no-resolv" "$config_file" # do not read /etc/resolv.conf
# use ctrld as upstream
pc_delete "server=" "$config_file"
pc_append "server=127.0.0.1#5354" "$config_file"
pc_append "add-mac" "$config_file" # add client mac
pc_delete "dnssec" "$config_file" # disable DNSSEC
pc_delete "trust-anchor=" "$config_file" # disable DNSSEC
# For John fork
pc_delete "resolv-file" "$config_file" # no WAN DNS settings
# Change /etc/resolv.conf, which may be changed by WAN DNS setup
pc_delete "nameserver" /etc/resolv.conf
pc_append "nameserver 127.0.0.1" /etc/resolv.conf
exit 0
fi
# GENERATED BY ctrld - EOF
On routers with nvram utility (ddwrt, freshtomato), you may want to check if the config was set correctly in router memory:
nvram show | grep dnsmasq_
If the service doesn't start, or hangs, try running it in debug mode and collect a log. This can be done using the following command:
ctrld start --log /tmp/ctrld.log -vv
The log file should contain some useful data. Open an issue with this data, and everything you collected above.