Accessing k8s secrets programmatically

[nheitz]

Hello all. I will freely admit I am still relatively new to the Rust ecosystem, and best described as a k8s "user" rather than power user or administrator, so please excuse any wild incompetence.

I built a small rust programme that retrieves oauth tokens for me from specific services in different environments, all of which are hosted on remote k8s clusters. Up until now, I've had to occasionally retrieve specific k8s secrets, and have done so by shelling out to kubectl and piping through jq. I generally don't like this kind of approach when more powerful integration is possible, however, hence my exploration of this project.

Whilst I have found querying for for pods and configmaps to be straightforward, querying for Secrets does not seems to work. Where a simple command line like

kubectl -n my-namespace get secrets

successfully returns a list of secrets,

What I think is the equivalent rust code:

let kube_config = Kubeconfig::from_env().unwrap().unwrap();
let options = KubeConfigOptions::default();
let mut config = Config::from_custom_kubeconfig(kube_config, &options).await.unwrap();
config.accept_invalid_certs = true;
let client = Client::try_from(config).unwrap();
let secrets: Api<Secret> = Api::namespaced(client, "my-namespace");
let lp = ListParams::default();
let result = secrets.list(&lp).await;

returns

[src/bin/test-k8s.rs:44] err = SerdeError(
    Error("invalid type: null, expected a base64-encoded string", line: 1, column: 38112),
)

I had to accept invalid certs for any of the queries to work, as the dev environment is not using an up to date cert.

My gut tells me that getting secrets requires an elevated level of access that is somehow transparent to the command line, but somehow I am not negotiating the protocol correctly in code.

Any ideas? Help would be much appreciated.

[nightkr]

Looks like some Secret has a data item with a null value? You should be able to find the offender with kubectl get secret -o json | jq '.items[] | {secret: .metadata.name, item: .data | to_entries | .[]} | select(.item.value == null)'.

Which version is your K8s cluster? This doesn't seem to be possible to reproduce with v1.19.1 at least, null values are implicitly converted to "".

[clux]

My gut tells me that getting secrets requires an elevated level of access that is somehow transparent to the command line, but somehow I am not negotiating the protocol correctly in code.

You should at least have enough access if you are receiving an object and gotten that far.
You can try working from the secret_reflector example and run NAMESPACE=somens cargo run --example secret_reflector in the examples directory.

[nheitz]

Looks like some Secret has a data item with a null value? You should be able to find the offender with kubectl get secret -o json | jq '.items[] | {secret: .metadata.name, item: .data | to_entries | .[]} | select(.item.value == null)'.

Which version is your K8s cluster? This doesn't seem to be possible to reproduce with v1.19.1 at least, null values are implicitly converted to "".

thank you both for such quick comments!

There is, indeed a "null" value lurking in the secrets...I am unfortunately still stuck on Server version v1.11.10. I did specify

k8s-openapi = { version = "0.11.0", default-features = false, features = ["v1_11"] }

but this does not seem to impact the issue. Is there possibly a bug here for 1_11 support, or shall I just accept this and revert to "shelling out" to kubectl until a server update in the future?

[nheitz]

My gut tells me that getting secrets requires an elevated level of access that is somehow transparent to the command line, but somehow I am not negotiating the protocol correctly in code.

You should at least have enough access if you are receiving an object and gotten that far.
You can try working from the secret_reflector example and run NAMESPACE=somens cargo run --example secret_reflector in the examples directory.

Before I raised this discussion, I did indeed attempt to run the reflector example, but it was no more able to get beyond this null value than my own simpler attempt.

[nightkr]

This seems to be a Kube apiserver issue. If you updated your cluster from an older version then it might have been fixed in your current version.. In that case you can try rewriting all Secrets to normalize them. To do that, run kubectl get secret --all-namespaces=1 --output=yaml | kubectl replace --file=-.

Sadly, Kind doesn't seem to support 1.11.x, so I can't really spin up a cluster and verify it myself.