kubernetes provider auth with client-go credential plugins such as azure kubelogin

[jludvice]

Hi, could anyone please advice how to configure kubernetes provider in a way to support
authentication with azure service principal through https://github.com/Azure/kubelogin ?

Thanks for any advice.

apiVersion: spinnaker.io/v1alpha2
kind: SpinnakerService
metadata:
  name: spinnaker
spec:
  spinnakerConfig:
    config:
      version: 1.22.1   # the version of Spinnaker to be deployed
      persistentStorage:
        persistentStoreType: azs
        azs:
          # ...
      providers:
        kubernetes:
          enabled: true
          accounts:
            - name: dev
              kubeconfigFile: dev-kube-config
# ...
          primaryAccount: dev
# ...
    files:
      dev-kube-config: |
        apiVersion: v1 ...
# ...
        users:
        - name:asdfasdfa
          user:
            exec:
              apiVersion: client.authentication.k8s.io/v1beta1
              args:
              - get-token
              - --environment
              - AzurePublicCloud
              - --server-id
              - asdfasdf
              - --client-id
              - asdfasdf
              - --tenant-id
              - asdasdf
              - --login
              - spn
              command: kubelogin
              env: null

I'm getting understandable message complaining that it can't find the kubelogin binary on $PATH

Resource: "spinnaker.io/v1alpha2, Resource=spinnakerservices", GroupVersionKind: "spinnaker.io/v1alpha2, Kind=SpinnakerService"
Name: "spinnaker", Namespace: "spinnaker"
for: "spinnakerservice-pepa.yml": admission webhook "webhook-spinnakerservices-v1alpha2.spinnaker.io" denied the request: 
SpinnakerService validation failed:
Validator for account 'dev' detected an error:
  error listing namespaces in account "dev":
  Get https://dev-3755ffd4.hcp.westeurope.azmk8s.io:443/api/v1/namespaces: getting credentials: exec: exec: "kubelogin": executable file not found in $PATH

Which pod is responsible for this validation ?

Can it be worked around by building custom image (based on yours) and adding the kubelogin binary in this custom image?

[jasonmcintosh]

Operator runs validation before deploying - and it's a combination of operator and halyard. Halyard is what's actually doing the validation. You could extend that to add it OR use a custom init container kinda things. Several different options depending upon need. Note you're LIKELY to hit some issues with spinnaker clouddriver as I don't think the kubelogin binary is on it either. Same sorta solution though - can use a custom init container.

Note, your kubeconfig would likely need to be updated to point to the custom mount of binaries (or as you mentioned just extend it). Can do a PR to OSS to try get kubeconfig to the docker images. https://github.com/Azure/kubelogin. Looking at this, SOUNDS like newer kubectls actually have everything kubeconfig did, but maybe not?

[jasonmcintosh]

Note not entirely sure a custom location in the kubconfig file will work :( but worth a shot...

[jludvice]

Thanks for insights @jasonmcintosh
For the time being I was able to work around this by creating service account in k8s and use those credentials for spinnaker.

So the idea is to get azure service principal login working is to get kubelogin to these two containers ?

• spinnaker-operator / halyard container
• spinnaker / spin-clouddriver pod

I'm not entirely sure I understand this:

Can do a PR to OSS to try get kubeconfig to the docker images. https://github.com/Azure/kubelogin. Looking at this, SOUNDS like newer kubectls actually have everything kubeconfig did, but maybe not?

Is it really possible to get binary to the container using initContainer?
The kubelogin binary size is over 1 MB

[jasonmcintosh]

https://kb.armory.io/s/article/Upgrade-Your-Kustomize-Version
This is an example of where this has been done for kustomize to use an overridden binary ;). Can do something similar for other services.