Having ServiceLB select the right IP/interface

[b4stien]

Hi everyone, as many before I have difficulties getting the real "client IP" in my pods receiving HTTP traffic. The only specifics of my setup is that my nodes are talking to each other over a pre-existing tailscale network. Everything except this "source client IP" works well.

I am well aware of externalTrafficPolicy: Local, which works well if I don't have a cluster communicating over Tailscale. I am also aware of the native k3s integration with Tailscale, I'd rather not use it as the machine is using the Tailscale network for unrelated stuff.

My install line is:

curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION='v1.31.5+k3s1' INSTALL_K3S_EXEC='server --cluster-init --flannel-iface tailscale0 --node-ip <tailscale IP>' sh -

Resulting in:

> kubectl get nodes -o wide                                                                                                                                                                                                         
NAME                  STATUS   ROLES                       AGE   VERSION        INTERNAL-IP     EXTERNAL-IP   OS-IMAGE                         KERNEL-VERSION         CONTAINER-RUNTIME
test-server-01   Ready    control-plane,etcd,master   89m   v1.31.5+k3s1   100.82.255.73   <none>        Debian GNU/Linux 12 (bookworm)   6.1.0-31-cloud-arm64   containerd://1.7.23-k3s2

The resulting problem is that the built-in Service LB is handing the built-in Traefik service the "wrong IP":

> kubectl get services -o wide --all-namespaces | grep traefik                                                                                                                                                                      
kube-system    traefik                        LoadBalancer   10.43.71.28     100.82.255.73   80:31263/TCP,443:30774/TCP   90m   app.kubernetes.io/instance=traefik-kube-system,app.kubernetes.io/name=traefik

Which then results in my downstream pods receiving: "x-forwarded-for":"10.42.0.11, 10.42.0.8" instead of the real client IP.

I'm guessing if I could tell Service LB to hand over my "real public IP" instead of the Tailscale internal one it would improve the situation. Except setting the External IP directly on the nodes is causing other issues.

Is there a way to tell Service LB which IP/interface to use and not have it select the internal one?

(I am using the last version of k3s)

[brandond]

ServiceLB is not aware of interfaces or anything else. All the pods do is bind to ports on the host (not to any specific address or interface), and forward traffic from there to the service address and ports. The service controller itself advertises the IPs of the Kubernetes nodes. If you want to use a different IP, you can override the node-ip or node-external-ip - but this will just change what shows up as the LoadBalancer IP, it won't actually change how anything listens.

[b4stien]

Understood.

What I do not understand is that the exact same config without telling flannel to use the tailscale0 works perfectly, with me getting "x-forwarded-for":"<Proper Client IP>, 10.42.0.8".

In this setup I have:

> kubectl get nodes -o wide                                                                                                                                                                                   
NAME                  STATUS   ROLES                       AGE   VERSION        INTERNAL-IP     EXTERNAL-IP   OS-IMAGE                         KERNEL-VERSION         CONTAINER-RUNTIME
dsit2-k3s-server-60   Ready    control-plane,etcd,master   11m   v1.31.5+k3s1   172.31.12.196   <none>        Debian GNU/Linux 12 (bookworm)   6.1.0-31-cloud-arm64   containerd://1.7.23-k3s

And:

> kubectl get services -o wide --all-namespaces | grep traefik                                                                                                                                                
kube-system    traefik                        LoadBalancer   10.43.129.26    172.31.12.196   80:31838/TCP,443:30423/TCP   11m     app.kubernetes.io/instance=traefik-kube-system,app.kubernetes.io/name=traefik

What would make this setup works with a K8S "Internal IP" being the private IP of the machine given by its hypervisor (it's and EC2 VM) and not work with a K8S "Internal IP" being the private IP given by Tailscale?

[brandond]

Generally speaking, any time client traffic comes in through an address translation layer that uses SNAT to ensure that return traffic goes back out the correct path, you will loose the original source address. You'd have to look at the inbound packet path and any iptables rules or other SNAT applied by your node, or by tailscale.

[b4stien]

Generally speaking, any time client traffic comes in through an address translation layer that uses SNAT to ensure that return traffic goes back out the correct path, you will loose the original source address.

Yes, I get that, but the whole point of setting externalTrafficPolicy: Local on the traefik service is to ensure there is not SNAT, right?

You'd have to look at the inbound packet path and any iptables rules or other SNAT applied by your node, or by tailscale.

That's what I don't get, in both cases (wether flannel is set to tailscale0 or not) the inbound packet is coming from the same interface, as the inbound packet has the public IP of the VM as its destination and is coming from the internet.

[brandond]

externalTrafficPolicy: Local just modifies the iptables rule in the pod to send traffic directly to the local node IP and nodePort, instead of to the service ClusterIP and port. Doing this prevents the traffic from being affected by kube-proxy's SNAT when transiting to a pod on another node. This of course requires that there be a pod for the service running locally.