feat: automate GCP Workload Identity via Config Connector IAMPolicyMember by lukebond · Pull Request #102 · restatedev/restate-operator

lukebond · 2026-03-18T13:23:54Z

this is basically the GCP equivalent of what we're doing for AWS with PodIdentityAssociations.

when serviceAccountAnnotations contains iam.gke.io/gcp-service-account,
the operator creates an IAMPolicyMember CR to bind the K8s service account
to the GCP service account via roles/iam.workloadIdentityUser
GCP project ID is extracted from the SA email
if the IAMPolicyMember CRD is absent (Config Connector not installed), the
operator sets a clear NotReady status rather than crashing
canary Job validates Workload Identity token from the GKE metadata server
before allowing the StatefulSet to proceed
extracted shared CanaryConfig, canary_job_spec, and check_canary_conditions
helpers to deduplicate canary logic between AWS and GCP
added IAMPolicyMember CRD types (hand-written minimal subset)
added conditional IAMPolicyMember and canary Job watchers in the controller
added unconditional RBAC rules for iampolicymembers.iam.cnrm.cloud.google.com
new unit tests covering GCP helpers and shared canary logic
added release notes

i tested this by:

creating a kind cluster and verifying the graceful failure of the IAMPolicyMember creation when CRD isn't present
created a GKE cluster with the Config Connect add-on and tested that:
- operator detects iam.gke.io/gcp-service-account in serviceAccountAnnotations and creates an IAMPolicyMember CR with correct member format (serviceAccount:PROJECT.svc.id.goog[NS/restate]), role
  (roles/iam.workloadIdentityUser), and resource ref (projects/PROJECT/serviceAccounts/EMAIL)
- Config Connector syncs the IAMPolicyMember and creates the IAM binding in GCP
- canary job obtains a Workload Identity token from the GKE metadata server, confirming the binding works end-to-end
- operator gates StatefulSet creation on canary success, then marks RestateCluster as Ready
- when Config Connector is not installed (kind cluster), operator sets NotReady with message "IAMPolicyMember CRD not found - is Config Connector installed?" rather than crashing
- removing the annotation returns the RestateCluster to Ready and removes the annotation from the ServiceAccount
- adding a bunch of unit tests to cover helper functions and shared canary logic

…mber - when serviceAccountAnnotations contains iam.gke.io/gcp-service-account, the operator creates an IAMPolicyMember CR to bind the K8s service account to the GCP service account via roles/iam.workloadIdentityUser - GCP project ID is extracted from the SA email - if the IAMPolicyMember CRD is absent (Config Connector not installed), the operator sets a clear NotReady status rather than crashing - canary Job validates Workload Identity token from the GKE metadata server before allowing the StatefulSet to proceed - extracted shared CanaryConfig, canary_job_spec, and check_canary_conditions helpers to deduplicate canary logic between AWS and GCP - added IAMPolicyMember CRD types (hand-written minimal subset) - added conditional IAMPolicyMember and canary Job watchers in the controller - added unconditional RBAC rules for iampolicymembers.iam.cnrm.cloud.google.com - new unit tests covering GCP helpers and shared canary logic - added release notes

- run_canary_job now returns Result<CanaryResult, Error> instead of Result<(), Error> so callers can tell if the job succeeded or is still running - fixes a bug where a completed canary was treated as pending, causing the operator to loop indefinitely on NotReady - also fixes Config Connector IAMServiceAccount external ref format to use projects/PROJECT/serviceAccounts/EMAIL this appears to touch AWS code but it's only due to them sharing some common code, it's just a return type change.

pcholakov

Nice work, @lukebond! Some non-blocking comments from the peanut gallery. :-)

pcholakov · 2026-03-18T14:32:08Z

charts/restate-operator-helm/templates/rbac.yaml

+  - resources:
+      - iampolicymembers
+    verbs:
+      - get
+      - list
+      - watch
+      - create
+      - patch
+      - delete
+    apiGroups:
+      - iam.cnrm.cloud.google.com


PodIdentityAssociation RBAC is conditional on .Values.awsPodIdentityAssociationCluster. Is it worth introducing a similar flag (e.g., .Values.gcpConfigConnector) for GCP? I believe the rules are harmless when the CRD doesn't exist so this is largely cosmetic.

i did test that it was harmless and therefore left it in. but you're right we should do the same.

pcholakov · 2026-03-18T14:37:49Z

src/controllers/restatecluster/reconcilers/compute.rs

+        .and_then(|s| s.service_account_annotations.as_ref())
+        .and_then(|a| a.get("iam.gke.io/gcp-service-account"));
+
+    if let Some(gcp_sa_email) = gcp_sa_email {


We don't clean up removed iam.gke.io/gcp-service-account annotations. The AWS PIA block explicitly deletes the PodIdentityAssociation and canary job. As is, I think removing the annotation would leave IAMPolicyMember and the restate-wi-canary job orphaned. Does Config Connector handle IAMPolicyMember removal if not referenced? Either way, we should probably delete the job if sa is unset.

} else { delete_job(name, &job_api, "restate-wi-canary").await?; // also IAMPolicyMember cleanup if necessary }

the operator owns the job and IAMPolicyMember via k8s owner references, so would be cleaned up when the parent CR is removed. i figured that was enough, wdyt?

actually i think you're right. i'll clean it up

pcholakov · 2026-03-18T14:40:36Z

src/controllers/restatecluster/reconcilers/compute.rs

+    // The eks-pod-identity-token volume is visible immediately at pod creation.
+    let name = config.name;
+    if let Ok(pods) = pod_api
+        .list(&ListParams::default().labels(&format!("batch.kubernetes.io/job-name={name}")))


I don't quite understand the implications of removing the UID filter, but the code originally included batch.kubernetes.io/controller-uid={} - was it intentional? Is there a difference in behavior?

this was a casualty of the refactor to reuse most of the existing job code so it could be shared, mostly, across providers. the UID wasn't available in the shared code path post-refactor. if there were somehow two jobs with the same name it might grab the wrong one but since no two in the same namespace can have the same name, i think it's fine. might be worth a clarifying comment though perhaps?

pcholakov · 2026-03-18T14:43:43Z

src/controllers/restatecluster/reconcilers/compute.rs

+    if let Some(status) = &ipm.status
+        && let Some(conditions) = &status.conditions
+        && let Some(ready) = conditions.iter().find(|cond| cond.r#type == "Ready")
+        && ready.status == "True"
+    {
+        true
+    } else {
+        false
+    }


Nit: I think this can just be:

Suggested change

if let Some(status) = &ipm.status

&& let Some(conditions) = &status.conditions

&& let Some(ready) = conditions.iter().find(|cond| cond.r#type == "Ready")

&& ready.status == "True"

{

true

} else {

false

}

ipm.status

.as_ref()

.and_then(|s| s.conditions.as_ref())

.and_then(|cs| cs.iter().find(|c| c.r#type == "Ready"))

.is_some_and(|c| c.status == "True")

nice, thanks! i'll do this

- added cleanup of IAMPolicyMember and canary Job when annotation is removed - added comment explaining why UID filter isn't needed on pod lookup - simplified is_iam_policy_member_ready to functional chain style - made IAMPolicyMember RBAC conditional on gcpConfigConnector Helm value

lukebond added 2 commits March 17, 2026 18:12

lukebond requested review from pcholakov and tillrohrmann March 18, 2026 13:23

pcholakov reviewed Mar 18, 2026

View reviewed changes

pcholakov approved these changes Mar 18, 2026

View reviewed changes

lukebond merged commit 2575a38 into main Mar 18, 2026
2 checks passed

github-actions bot locked and limited conversation to collaborators Mar 18, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: automate GCP Workload Identity via Config Connector IAMPolicyMember#102

feat: automate GCP Workload Identity via Config Connector IAMPolicyMember#102
lukebond merged 3 commits intomainfrom
feat/gcp-workload-identity-configconnector

lukebond commented Mar 18, 2026 •

edited

Loading

Uh oh!

pcholakov left a comment

Uh oh!

pcholakov Mar 18, 2026

Uh oh!

lukebond Mar 18, 2026

Uh oh!

pcholakov Mar 18, 2026

Uh oh!

lukebond Mar 18, 2026

Uh oh!

lukebond Mar 18, 2026

Uh oh!

pcholakov Mar 18, 2026

Uh oh!

lukebond Mar 18, 2026

Uh oh!

pcholakov Mar 18, 2026

Uh oh!

lukebond Mar 18, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

lukebond commented Mar 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pcholakov left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

lukebond commented Mar 18, 2026 •

edited

Loading