Struggling to switch from servicelb to metallb (no access from the outside)

[belugame]

Hey all,

I have k3s installed on a Debian virtual machine where I sadly do not have access to more information regarding the surrounding setup except that it is behind a NAT. I get to it via ssh a.b.c.d. And there is mydomain.com pointing to it.

Initially I was using a default k3s install with servicelb included and I was able to reach my container via my config for traefik/ingressroute with host-matching/service/pod/container on port 80 and 443 from the outside world via mydomain.com. So far so good.

One hard requirement is that I am able to see the source IP of requests in my containers somehow and I could not get it to work with traefik except with hostNetwork: true which from what I read is discouraged due to security implications and limiting functionality in a multi node setup (we have 2 more nodes waiting to be included).

So I reinstalled K3S with v1.32.2-rc1+k3s1 to already have traefik v3 and --disable servicelb, and added metallb (configs below).

I can see my traefik service receive an external ip from the range i specify. When I use an internal range 192.168.1.240 I can curl it internally from the host and see the container's response. But when I specify a.b.c.d, I cannot reach it at all. nc -zv a.b.c.d no longer reports port 80/443 as open from the outside.

With servicelb I did not have to specify a.b.c.d at all and it as far as I understood did the iptables for me. Now with netstat I don't see the host listening on port 80 or 443. And don't know where I am going wrong and what are the missing pieces. If I need to use BGP or L2 with this NAT-setup I have.

When I do ip addr show I do not see a.b.c.d and ip route show default gives me default via 10.5.1.1 dev ens192 onlink.

          apiVersion: metallb.io/v1beta1
          kind: IPAddressPool
          metadata:
            name: public-ip-pool
            namespace: metallb-system
          spec:
            addresses:
            - 192.168.1.240-192.168.1.250  # I'm unsure if i need to use a.b.c.d here
          ---
          apiVersion: metallb.io/v1beta1
          kind: L2Advertisement
          metadata:
            name: metallb-advertisment
            namespace: metallb-system
          spec:
            ipAddressPools:
            - public-ip-pool

My traefik:

          apiVersion: helm.cattle.io/v1
          kind: HelmChartConfig
          metadata:
            name: traefik
            namespace: kube-system
          spec:
            valuesContent: |-
              type: LoadBalancer
              additionalArguments:
                - "--entryPoints.web.address=:80/tcp"
                - "--entryPoints.websecure.address=:443/tcp"
              logs:
                general:
                  level: DEBUG
                access:
                  addinternals: false
                  enabled: true
                  format: json
                  fields:
                    defaultmode: keep
                    headers:
                      defaultmode: keep
                      names:
                        X-Forwarded-For: keep
                        X-Real-Ip: keep
                        BackendAddr: keep
                        ClientHost: keep
                        ClientAddr: keep
                        RequestAddr: keep
              entryPoints:
                web:
                  address: ":80"
                websecure:
                  address: ":443"

              ports:
                web:
                  port: 80
                  expose:
                    default: true
                  protocol: TCP
                  exposedPort: 80
                websecure:
                  protocol: TCP
                  port: 443
                  expose:
                    default: true
                  exposedPort: 443

The service:

          apiVersion: v1
          kind: Service
          metadata:
            name: traefik
            namespace: kube-system
          spec:
            type: LoadBalancer
            selector:
              app.kubernetes.io/name: traefik
            ports:
              - name: web
                port: 80
                targetPort: 80
                protocol: TCP
              - name: websecure
                port: 443
                targetPort: 443
                protocol: TCP