Use projected token volume for hostNetwork pods. #428

siyanshen · 2025-01-14T18:55:22Z

Why we need this change

Before this change, GCSFuse CSI driver does not support pods with hostNetwork:true. This is because the gcsfuse process runs as part of a sidecar container that is injected into the user pod. GCSFuse uses the ADC workflow to fetch the necessary token to access a GCS bucket. However, with hostNetwork enabled, GKE metadata server cannot intercept the token requests for GET /computeMetadata/v1/instance/service-accounts/default/token endpoint.

What is in this change

For HostNetwork=true user pods, GCSFuse CSI webhook injects a projected SA token volume to the user pod.
gke-gcsfuse-sidecar container prepares a unix domain socket and starts a handler to serve requests on this token; invoke gcsfuse with a config option to point to the socket gcs-auth:token-url:<path to the token>
Token request handler in the gke-gcsfuse-sidecar handles token request from GCSFuse.

Local set up and testing

Test cases covered

pod with hostNetwork=true: can I/O GCS bucket.
pod with hostNetwork=false: is not affected and can I/O GCS bucket.
pod with hostNetwork=true, with GCS bucket A mounted as ephemeral storage and Gcs bucket B mounted as PV: can access bucket A & B without issue.

Setup

Create a cluster with workload identity pool enabled & GCSFuse disabled.
Grant GCS permissions to an SA

kubectl create namespace <ns>
kubectl create serviceaccount <test-ksa-ns> \
    --namespace <ns>

gcloud storage buckets add-iam-policy-binding gs://<bucket> \
    --member "principal://iam.googleapis.com/projects/<project-number>/locations/global/workloadIdentityPools/<project-id>.svc.id.goog/subject/ns/<ns>/sa/<test-ksa-ns>" \
    --role "roles/storage.objectUser"

Build and install your GCSFuse image

make build-image-and-push-multi-arch REGISTRY=<your-registry> STAGINGVERSION=<your-dev-tag>
make install REGISTRY=<your-registry> PROJECT=<project-id> STAGINGVERSION=<your-dev-tag>

Create a user pod with hostNetwork=true, GCS bucket mounted as a volume, and service account set as the one you created in step 2. Check I/O to your bucket from a container in your pod.

deploy/base/webhook/deployment.yaml

cmd/sidecar_mounter/main.go

pkg/sidecar_mounter/sidecar_mounter_config.go

cmd/sidecar_mounter/main.go

cmd/webhook/main.go

pkg/cloud_provider/auth/token_sources.go

pkg/sidecar_mounter/sidecar_mounter.go

hime · 2025-01-27T18:34:13Z

pkg/webhook/mutatingwebhook.go

@@ -110,6 +113,15 @@ func (si *SidecarInjector) Handle(_ context.Context, req admission.Request) admi

 	// Inject container.
 	injectSidecarContainer(pod, config, injectAsNativeSidecar)
+
+	if pod.Spec.HostNetwork {


Are there any cases in which the cx would want to keep using the GCE SA? (even though its not very safe)

Is that a common use case? This implementation will not be able to accommodate that

hime · 2025-01-27T18:37:43Z

pkg/webhook/mutatingwebhook.go

+		if err != nil {
+			return admission.Errored(http.StatusInternalServerError, fmt.Errorf("failed to get project id: %w", err))
+		}
+		pod.Spec.Volumes = append(pod.Spec.Volumes, GetSATokenVolume(projectID))


do you think we should have the logic that injects volumes in one place?

Could you elaborate?

pkg/webhook/sidecar_spec.go

hime · 2025-01-27T18:44:23Z

pkg/webhook/sidecar_spec.go

-	NobodyGID = 65534
+	NobodyUID           = 65534
+	NobodyGID           = 65534
+	tokenExpiryDuration = 600


Curious, why 600? Is tokenExpiryDuration something that can be adjusted? if so, should the cx have the ability to adjust?

Prolonged it to 3600-should be a more reasonable token expiry duration. See more about what this value does: https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/#launch-a-pod-using-service-account-token-projection

It's not very useful to make tokenExpiryDuration configurable.

When this expires, we have the ability to renew because of the StartTokenServer call. Do we plan to add e2e tests in a followup PR to test the functionality of the token server? If we have a TokenServer, do we favor a shorter or longer tokenExpiryDuration?

I plan to add an e2e test to test GCS bucket access with hostnetwork enabled pods. That should cover the functionality of the token server.

RetokenExpiryDuration- we will have to weigh in the security benefits of shorter token lifetimes against the performance benefits of longer lifetimes. Kubernetes' approach provides a good starting point- it will refresh when it hits 80% of TTL or 24h (whichever is shorter). I think 3600s(1h) is a nice balance point between security and network traffic overhead.

pkg/webhook/sidecar_spec.go

pkg/sidecar_mounter/sidecar_mounter.go

kislaykishore · 2025-01-27T18:21:00Z

pkg/sidecar_mounter/sidecar_mounter.go

+		WriteTimeout: 10 * time.Second,
+	}
+
+	if err := server.Serve(socket); err != nil {


Is there a way to stop the server gracefully during shutdown?

This implementation follows the standard pattern across the repo. e.g.: metrics server:

gcs-fuse-csi-driver/pkg/sidecar_mounter/sidecar_mounter.go

Line 249 in e5d8871

if err := server.Serve(socket); err != nil {

Can you give me an example of gracefully shutting down a server? Is there a significant benefit for doing that?

pkg/sidecar_mounter/sidecar_mounter.go

pkg/sidecar_mounter/token_manager.go

cmd/webhook/main.go

kislaykishore · 2025-01-27T19:01:29Z

pkg/sidecar_mounter/sidecar_mounter.go

+	return audience, nil
+}
+
+func StartTokenServer(ctx context.Context) {


What's the behavior when the server failed to start? Will the GCS calls use node's auth scopes?

The pod will not have access to GCS bucket in this case

And mounting will fail.

pkg/sidecar_mounter/token_source.go

pkg/sidecar_mounter/sidecar_mounter.go

pkg/sidecar_mounter/sidecar_mounter_config.go

pkg/sidecar_mounter/sidecar_mounter.go

pkg/sidecar_mounter/sidecar_mounter_config_test.go

siyanshen force-pushed the hostnetwork branch 7 times, most recently from 3b2b99f to 557c193 Compare January 15, 2025 23:56

siyanshen requested review from hime, saikat-royc, mattcary, msau42, kislaykishore and Tulsishah and removed request for saikat-royc January 16, 2025 00:32

mattcary reviewed Jan 22, 2025

View reviewed changes

deploy/base/webhook/deployment.yaml Outdated Show resolved Hide resolved

cmd/sidecar_mounter/main.go Outdated Show resolved Hide resolved

pkg/sidecar_mounter/sidecar_mounter_config.go Outdated Show resolved Hide resolved

cmd/sidecar_mounter/main.go Outdated Show resolved Hide resolved

siyanshen force-pushed the hostnetwork branch 2 times, most recently from 9afba55 to 2f4c6e4 Compare January 23, 2025 00:47

siyanshen removed the request for review from msau42 January 27, 2025 17:45