Failing to connect to https://registry.k8s.io/ from inside the minikube container - MINIKUBE_ROOTLESS=true

[gilbertoca]

What Happened?

Following the docs on using podman in rootless mode, I've execute:

gilberto.andrade@C430760:~$ minikube config set rootless true
gilberto.andrade@C430760:~$ minikube start --driver=podman --container-runtime=containerd
😄  minikube v1.35.0 on Opensuse-Leap 15.6
    ▪ MINIKUBE_ROOTLESS=true
✨  Using the podman driver based on user configuration
📌  Using rootless Podman driver
👍  Starting "minikube" primary control-plane node in "minikube" cluster
🚜  Pulling base image v0.0.46 ...
💾  Downloading Kubernetes v1.32.0 preload ...
    > preloaded-images-k8s-v18-v1...:  379.64 MiB / 379.64 MiB  100.00% 29.61 M
    > gcr.io/k8s-minikube/kicbase...:  500.31 MiB / 500.31 MiB  100.00% 28.55 M
E0225 13:45:30.254467   25543 cache.go:222] Error downloading kic artifacts:  not yet implemented, see issue #8426
🔥  Creating podman container (CPUs=2, Memory=3900MB) ...
❗  Failing to connect to https://registry.k8s.io/ from inside the minikube container
💡  To pull new external images, you may need to configure a proxy: https://minikube.sigs.k8s.io/docs/reference/networking/proxy/
📦  Preparing Kubernetes v1.32.0 on containerd 1.7.24 ...
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔗  Configuring CNI (Container Networking Interface) ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: storage-provisioner, default-storageclass
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

Checking:

gilberto.andrade@C430760:~$ kubectl get po -A
NAMESPACE     NAME                               READY   STATUS    RESTARTS      AGE
kube-system   coredns-668d6bf9bc-zcdck           1/1     Running   0             33m
kube-system   etcd-minikube                      1/1     Running   0             33m
kube-system   kindnet-ljs2g                      1/1     Running   0             33m
kube-system   kube-apiserver-minikube            1/1     Running   0             33m
kube-system   kube-controller-manager-minikube   1/1     Running   0             33m
kube-system   kube-proxy-bj5j9                   1/1     Running   0             33m
kube-system   kube-scheduler-minikube            1/1     Running   0             33m
kube-system   storage-provisioner                1/1     Running   1 (33m ago)   33m

gilberto.andrade@C430760:~$ cat /etc/os-release 
NAME="openSUSE Leap"
VERSION="15.6"

gilberto.andrade@C430760:~$ minikube version
minikube version: v1.35.0
commit: dd5d320e41b5451cdf3c01891bc4e13d189586ed-dirty

I've noted that it complained "Failing to connect to https://registry.k8s.io/ from inside the minikube container" and so I've tried to pull an image from inside:

gilberto.andrade@C430760:~$ minikube ssh
docker@minikube:~$ podman version
WARN[0000] The cgroupv2 manager is set to systemd but there is no systemd user session available 
WARN[0000] For using systemd, you may need to login using an user session 
WARN[0000] Alternatively, you can enable lingering with: `loginctl enable-linger 1000` (possibly as root) 
WARN[0000] Falling back to --cgroup-manager=cgroupfs    
Error: command required for rootless mode with multiple IDs: exec: "newuidmap": executable file not found in $PATH

docker@minikube:~$ sudo podman version
Version:      3.4.4
API Version:  3.4.4
Go Version:   go1.18.1
Built:        Thu Jan  1 00:00:00 1970
OS/Arch:      linux/amd64

docker@minikube:~$ sudo podman pull registry.access.redhat.com/ubi8/ubi
Trying to pull registry.access.redhat.com/ubi8/ubi:latest...
Error: initializing source docker://registry.access.redhat.com/ubi8/ubi:latest: pinging container registry registry.access.redhat.com: Get "https://registry.access.redhat.com/v2/": dial tcp: lookup registry.access.redhat.com on 192.168.49.1:53: read udp 192.168.49.2:59184->192.168.49.1:53: i/o timeout

docker@minikube:~$ exit
logout
ssh: Process exited with status 125

gilberto.andrade@C430760:~$ podman pull registry.access.redhat.com/ubi8/ubi
Trying to pull registry.access.redhat.com/ubi8/ubi:latest...
Getting image source signatures
Copying blob f8750fc49bf2 done   | 
Copying config dc564c6cc0 done   | 
Writing manifest to image destination
dc564c6cc036db0628baa2d1dd623054d115a2528c57b1cf5324ee9dedb019cb

gilberto.andrade@C430760:~$ 

Is it possível to update podman in minukube image? Maybe it is the problem?

Attach the log file

log.txt

Operating System

Other

Driver

Podman

[gilbertoca]

It seems the internal minikube dns is intermittent:

gilberto.andrade@C430760:~$ minikube ssh
docker@minikube:~$ nslookup registry-1.docker.io
;; communications error to 192.168.49.1#53: timed out
;; communications error to 192.168.49.1#53: timed out
Server:         192.168.49.1
Address:        192.168.49.1#53

Non-authoritative answer:
Name:   registry-1.docker.io
Address: 98.85.153.80
Name:   registry-1.docker.io
Address: 44.208.254.194
Name:   registry-1.docker.io
Address: 3.94.224.37
;; communications error to 192.168.49.1#53: timed out
Name:   registry-1.docker.io
Address: 2600:1f18:2148:bc01:f43d:e203:cafd:8307
Name:   registry-1.docker.io
Address: 2600:1f18:2148:bc00:5cac:48a0:7f88:7266
Name:   registry-1.docker.io
Address: 2600:1f18:2148:bc02:22:27bd:19a8:870c

docker@minikube:~$ 

I've tried to replace the internal one by:
1- editing kubectl edit configmap coredns -n kube-system - no effect;
2- editing podman

gilberto.andrade@C430760:~$ cat ~/.config/containers/containers.conf
[containers]
log_driver = "k8s-file"

[network]
dns_servers = ["8.8.8.8", "8.8.4.4"]
 

no effect;
3-editing the /etc/resolv.conf

gilberto.andrade@C430760:~$ minikube ssh
docker@minikube:~$ cat /etc/resolv.conf
search dns.podman
nameserver 192.168.49.1

docker@minikube:~$ sudo sh -c 'echo "nameserver 8.8.8.8" > /etc/resolv.conf'
docker@minikube:~$ sudo sh -c 'echo "nameserver 8.8.4.4" >> /etc/resolv.conf'
docker@minikube:~$ cat /etc/resolv.conf
nameserver 8.8.8.8
nameserver 8.8.4.4

this one made it finally works.

docker@minikube:~$ cat /etc/resolv.conf
nameserver 8.8.8.8
nameserver 8.8.4.4
docker@minikube:~$ nslookup registry-1.docker.io
Server:         8.8.8.8
Address:        8.8.8.8#53

Non-authoritative answer:
Name:   registry-1.docker.io
Address: 3.94.224.37
Name:   registry-1.docker.io
Address: 98.85.153.80
Name:   registry-1.docker.io
Address: 44.208.254.194
Name:   registry-1.docker.io
Address: 2600:1f18:2148:bc00:5cac:48a0:7f88:7266
Name:   registry-1.docker.io
Address: 2600:1f18:2148:bc01:f43d:e203:cafd:8307
Name:   registry-1.docker.io
Address: 2600:1f18:2148:bc02:22:27bd:19a8:870c