bug: CNPG instance manager trying to update credentials for non-existent `app` user

[artntek]

CNPG's default database owner name is app. When using a non-default name with the cnpg chart, the controller believes the app user still exists, and logs errors because it cannot rotate the credentials. Example from vegbank prod deployment cnpg logs:

{"level":"error","ts":"2026-03-30T17:06:40.552478459Z","msg":"Reconciler error",
"logger":"instance-manager","logging_pod":"vegbankdb2-cnpg-1",
"controller":"instance-cluster","controllerGroup":"postgresql.cnpg.io",
"controllerKind":"Cluster","Cluster":{"name":"vegbankdb2-cnpg",
"namespace":"vegbank"},"namespace":"vegbank","name":"vegbankdb2cnpg""reconcileID":"31cff6ce-c7fc-4912-835c-879dddd05fdf",
"error":"while updating database owner password: ERROR: role \"app\" does not exist (SQLSTATE 42704)",
"stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\tpkg/mod/sigs.k8s.io/controller-
runtime@v0.21.0/pkg/internal/controller/controller.go:353\nsigs.k8s.io/controller-
runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\tpkg/mod/sigs.k8s.io/controller-
runtime@v0.21.0/pkg/internal/controller/controller.go:300\nsigs.k8s.io/controller-
runtime/pkg/internal/controller.(*Controller[...]).Start.func2.1\n\tpkg/mod/sigs.k8s.io/controller-
runtime@v0.21.0/pkg/internal/controller/controller.go:202"}

{"level":"info","ts":"2026-03-30T17:06:40.552524335Z","logger":"postgres","msg":"record",
"logging_pod":"vegbankdb2-cnpg-1","record":{"log_time":"2026-03-30 17:06:40.552 UTC",
"user_name":"postgres","database_name":"postgres","process_id":"1443",
"connection_from":"[local]","session_id":"69caada0.5a3","session_line_num":"1",
"command_tag":"ALTER ROLE","session_start_time":"2026-03-30 17:06:40 UTC",
"virtual_transaction_id":"84/17","transaction_id":"0","error_severity":"ERROR",
"sql_state_code":"42704","message":"role \"app\" does not exist",
"query":"ALTER ROLE \"app\" WITH PASSWORD '(MB redacted)'",
"application_name":"cnpg-instance-manager","backend_type":"client backend","query_id":"0"}}

[artntek]

What's happening:

CNPG's instance manager has a built-in reconciliation loop that manages the app role (password rotation, ownership, etc.). Even when you configure a custom username, it still tries to manage the default app role unless you explicitly tell it not to.

How to fix it:

In your Cluster spec, make sure your custom username is set under bootstrap.initdb.owner (or bootstrap.pg_basebackup.owner) and that you're not inadvertently leaving the default app role expected. The key field is in your Cluster CR:

spec:
  bootstrap:
    initdb:
      database: vegbankdb2
      owner: your_custom_user   # <-- this must match your Secret
  managed:
    roles: []  # optionally suppress managed role reconciliation

Also check your bootstrap secret reference — CNPG derives the expected role name from the secret named in bootstrap.initdb.secret.name. If that secret has a username key, CNPG uses it. If it's missing or mismatched, it falls back to app.

Quick diagnostic questions:

• Does your Cluster spec have bootstrap.initdb.owner explicitly set to your custom username?
• Is the referenced secret's username field set correctly?
• Did this cluster get migrated or restored from a backup where the app role may never have been created?

The error will keep recurring on every reconcile loop until CNPG can either find the app role or you correct the owner configuration to match what actually exists in Postgres.

[artntek]

~/dev/GITHUB/NCEAS/vegbank2 (develop) ✔                                                  (⎈|prod-k8s:N/A)
 brooke@magnum-pi.local $ kubectl get pods -A \
         -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name --no-headers | \
         grep "cnpg-1" | while read -r ns pod; do
             echo "Checking $ns/$pod..."
             kubectl logs "$pod" -n "$ns" | grep "role ..app.. does not exist" | head -1 | cut -c 440-540
         done
Checking arctic/mcdbarctic-cnpg-1...
Defaulted container "postgres" out of: postgres, bootstrap-controller (init)
sword: ERROR: role \"app\" does not exist (SQLSTATE 42704)","stacktrace":"sigs.k8s.io/controller-runt
Checking esa/mcdbesa-cnpg-1...
Defaulted container "postgres" out of: postgres, bootstrap-controller (init)
saction_id":"0","error_severity":"ERROR","sql_state_code":"42704","message":"role \"app\" does not ex
Checking goa/mcdbgoa-cnpg-1...
Defaulted container "postgres" out of: postgres, bootstrap-controller (init)
Checking keycloak/keycloakx-cnpg-1...
Defaulted container "postgres" out of: postgres, bootstrap-controller (init)
Checking opc/mcdbopc-cnpg-1...
Defaulted container "postgres" out of: postgres, bootstrap-controller (init)
nsaction_id":"0","error_severity":"ERROR","sql_state_code":"42704","message":"role \"app\" does not e
Checking sdr/mcdbsdr-cnpg-1...
Defaulted container "postgres" out of: postgres, bootstrap-controller (init)
ransaction_id":"0","error_severity":"ERROR","sql_state_code":"42704","message":"role \"app\" does not
Checking vegbank/vegbankdb2-cnpg-1...
Defaulted container "postgres" out of: postgres, bootstrap-controller (init)
id":"0","error_severity":"ERROR","sql_state_code":"42704","message":"role \"app\" does not exist","qu