Static Private IP Possible?

[jordanabakerafs]

I am currently supporting a team which is using the kubernetes-ingress (not plus) to set up ingress on services, and one of the issues we're come across is the private IP of the ingress pods can potentially change if the cluster reboots/updates - which breaks DNS resolution.

We found some deprecated config values that seem to point to this - but nothing current. Is it possible to set a static private IP value for the ingress pods in the deployment.

Setup:

• Standard Nginx-Ingress
• Azure Kubernetes Service - GCC High
• Ingress is deployed into it's own namespace on our linux agentpool running CBL Mariner.

[haywoodsh]

Hi @jordanabakerafs can you tell us more why you need to access to the pods via their IP but not via the Kubernetes service?

When we deploy NGINX Ingress Controller on cloud environments, we can access the Ingress Controller through the load balancer provided by the cloud vendor, with the following kubectl command.

kubectl describe svc <release-name> -n <namespace> | grep "LoadBalancer Ingress"

For details on how to get the external IP, please refer to our documentation.
https://docs.nginx.com/nginx-ingress-controller/installation/installing-nic/installation-with-manifests/#how-to-access-nginx-ingress-controller

[jordanabakerafs]

Hi @haywoodsh - thanks for responding.

We have a requirement to deploy a private cluster with no external ingress. We require the ingress to be reachable on a private IP which would be sourced from the subnet tied to our nodepool. We're currently doing this by setting our DNS record manually to the the "Endpoints" value which is the internal POD ip. We recognize this is a brittle solution - though we've thus far been unable to figure out a way to set the ingress URL to an internal IP, and not an external public IP.

https://kubernetes.io/docs/concepts/services-networking/service/#loadbalancer - We were able to find this here, but it appears to be deprecated.

We are currently looking into this - https://learn.microsoft.com/en-us/azure/aks/create-nginx-ingress-private-controller, but we were trying to determine if there was a way to set a private address on our existing ingress without redeploying/configuring.

[j1m-ryan]

Hi @jordanabakerafs,
I think you have found the correct annotation with that azure docs link.

service.beta.kubernetes.io/azure-load-balancer-internal: "true"

I want to point out that the Ingress controller in those azure docs is the community ingress controller.

However this annotation will work for any ingress controller

If you are using helm you can add this annotation to the loadbalancer service in the NGINX Ingress Controller's helm chart

controller:
    service:
        annotations:
            service.beta.kubernetes.io/azure-load-balancer-internal: "true"

Let us know how you get on

[jordanabakerafs]

Thanks for the reply @j1m-ryan - we're going to do some testing this week and we'll update the discussion with some updates and an answer for future contributors!

[jordanabakerafs]

Following up here with the solution:

@j1m-ryan confirmed we were heading in the correct direction with the annotation. We're using oci, so we ended up adding the following to our helm upgrade call:

--set controller.service.type=LoadBalancer
--set controller.service.loadBalancerIP=${{ vars.INGRESS_INTERNAL_IP }}
--set-string controller.service.annotations."service\.beta\.kubernetes\.io/azure-load-balancer-internal"=true

Some additional items to note:

• The kubelet identity (we're using a user assigned managed identity) required 'Microsoft.Network/virtualNetworks/subnets/read' permissions over the target subnet. We applied the reader role for that MSI to the vnet, this was permissive enough
• The actual population of the internal IP took some time, our CI/CD pipeline wasn't expecting that so it timed out. Had to account for that.

After a helm uninstall to clear the loadBalancerIP, the deployment rolled out and we were able to see the internal IP under the "External IP of the cluster. After pointing our internal A record to this IP, we were able to successfully hit the cluster and reach our services.

Thanks @j1m-ryan for the assist!