[Design Proposal] Secret Management with Key Vault Integration

[hanzjk]

Problem

Agent deployments require configuration values to operate, but many of these values are sensitive (API keys, tokens, credentials, private endpoints). Treating configuration as plain values creates security risks and operational overhead. The platform currently lacks a secure, standardized mechanism to store, inject, and manage sensitive configurations, resulting in the following issues:

1. Plaintext exposure of sensitive data
2. Lack of secure runtime injection
3. No integration with enterprise secret managers
4. Risk of leaking secrets via logs and telemetry
5. Operational friction for secure production usage

User Stories

Platform Administrator

• As a platform administrator, I want to store and manage sensitive configuration values in a centralized secret store so that credentials are protected and comply with organizational security policies.
• As a platform administrator, I want the platform to integrate with a secret management system (e.g., Vault, OpenBao, cloud secret managers) so that we maintain a single source of truth and follow enterprise governance practices.
• As a platform administrator, I want audit trails and masking mechanisms so that secret usage is tracked while preventing exposure in logs and observability pipelines.
• As a platform administrator, I want to control which teams, projects, or environments can access specific secrets so that sensitive credentials are only available to authorized workloads.

Agent Developer

• As an agent developer, I want to reference secrets in my agent configuration without hardcoding sensitive values so that my code and configuration remain safe to share and version.
• As an agent developer, I want secrets to be injected automatically at runtime as environment variables or mounted files so that I do not need to implement custom credential handling.
• As an agent developer, I want to use different secrets for development, staging, and production environments so that I can deploy the same agent configuration safely across environments.
• As an agent developer, I want secret updates to take effect when configurations are updated.

Existing Solutions

N/A

Proposed Solution

Overview

This proposal introduces a secret management system that securely stores sensitive environment variables in OpenBao KV store and synchronizes them to Kubernetes using the External Secrets Operator. The system follows a provider-based architecture for extensibility and integrates with OpenChoreo via SecretReference custom resources.

Design

Architecture overview

flowchart TB
 subgraph AMS["Agent Manager Service"]
        STORE["1️⃣ Store secrets in OpenBao"]
        CREATE_SR["2️⃣ Create SecretReference CR"]
        CREATE_COMP["3️⃣ Create Component CR<br>(references ComponentType)"]
  end
 subgraph OpenBao["OpenBao KV Store"]
        KV["KV v2 Engine<br>Path: org/project/component"]
        META["Metadata<br>managed-by: amp-secret-manager"]
  end
 subgraph OC["OpenChoreo Control Plane"]
        SR["SecretReference CR"]
        CT["ComponentType<br>defines ExternalSecret template"]
        COMP["Component CR"]
  end
 subgraph DP["Data Plane"]
        ES["ExternalSecret CR"]
        K8S_SEC["K8s Secret"]
        POD["Workload Pod"]
  end
 subgraph ClusterWide["Cluster-Scoped"]
        CSS["ClusterSecretStore<br>(amp-openbao-store)"]
        ESO["External Secrets Operator"]
  end
    STORE -- 1️⃣ --> KV
    KV --> META
    CREATE_SR -- 2️⃣ --> SR
    CREATE_COMP -- 3️⃣ --> COMP
    COMP -. references .-> CT
    SR -. "remoteRef.key" .-> KV
    CT -- 4️⃣ template used to create --> ES
    SR -- 4️⃣ resolves secret paths --> ES
    ES -- 5️⃣ references --> CSS
    CSS -. configured to read .-> KV
    ESO -- 6️⃣ watches & processes --> ES
    ESO -- 7️⃣ fetches secrets via --> CSS
    ESO -- 8️⃣ creates --> K8S_SEC
    K8S_SEC -- 9️⃣ mounted as env --> POD

Loading

Secret Manager Client Interface and Registry

The secret management system follows a layered architecture with a provider registry pattern for extensibility.

flowchart TB
    subgraph HighLevel["High-Level Interface"]
        SMC["SecretManagementClient<br/>(CreateSecret, UpdateSecret, GetSecret, DeleteSecret)"]
    end

    subgraph Registry["Provider Registry"]
        REG["Registry<br/>(Thread-safe provider lookup)"]
        REG -->|"GetProvider('openbao')"| PI
    end

    subgraph Providers["Provider Layer"]
        PI["Provider Interface<br/>(NewClient, ValidateConfig, Capabilities)"]
        PI --> OBP["OpenBao Provider"]
        PI -.-> VP["Vault Provider (future)"]
        PI -.-> AWSP["AWS Secrets Manager (future)"]
    end

    subgraph LowLevel["Low-Level Operations"]
        OBC["OpenBao Client<br/>(KV v2 CRUD with ownership tracking)"]
    end

    SMC --> REG
    OBP --> OBC

Loading

• API surface (endpoints, schemas, protocols)
• Data model changes
• Component interactions

Layer 1: SecretManagementClient (High-Level Interface)

Provides domain-level CRUD operations with JSON marshaling/unmarshaling and error handling.

type SecretManagementClient interface {
    CreateSecret(ctx, req CreateSecretRequest) (*SecretResponse, error)
    UpdateSecret(ctx, orgName, path string, req UpdateSecretRequest) (*SecretResponse, error)
    GetSecret(ctx, orgName, path string) (*SecretResponse, error)
    DeleteSecret(ctx, orgName, path string) error
}

Layer 2: Provider Registry (Thread-Safe Lookup)

Enables pluggable KVs via runtime registration. Providers register themselves at init time.

// Providers register themselves at import time
func init() {
    secretmanagersvc.Register("openbao", NewProvider())
}

// Runtime lookup
provider, ok := secretmanagersvc.GetProvider("openbao")

Layer 3: Provider Interface (Factory Pattern)

Defines the contract for secret backend implementations.

type Provider interface {
    NewClient(ctx, config *StoreConfig) (SecretsClient, error)
    ValidateConfig(config *StoreConfig) error
    Capabilities() StoreCapabilities  // ReadOnly, WriteOnly, ReadWrite
}

Layer 4: SecretsClient (Low-Level Operations)

Provider-specific implementation for direct backend operations.

type SecretsClient interface {
    PushSecret(ctx, key string, value []byte, metadata *SecretMetadata) error
    GetSecret(ctx, key string) ([]byte, error)
    DeleteSecret(ctx, key string) error  // Only deletes if managed-by matches
}

OpenChoreo Integration Flow

The integration with OpenChoreo enables Kubernetes-native secret synchronization using SecretReference custom resources and the External Secrets Operator.

Step-by-Step Integration:

flowchart TB
 
        AMS["Agent Manager Service"]
        OB["OpenBao KV Store"]
        OC["OpenChoreo Controller"]
        SR["SecretReference CR"]
        ES["ExternalSecret CR"]
        COM["Component/Workload CR"]

        AMS -->|"1. Store secrets"| OB
        AMS -->|"2. Create"| SR
        AMS -->|"3. Create with SecretKeyRef"| COM
        OC -->|"4. Watch & Create"| ES
        SR -.->|"references KV path"| OB
   

        ESO["External Secrets Operator"]
        K8S["K8s Secret"]
        POD["Workload Pod"]

        ES -->|"5. Trigger"| ESO
        ESO -->|"6. Fetch from"| OB
        ESO -->|"7. Create/Update"| K8S
        K8S -->|"8. Mount as env"| POD

Loading

Step	Component	Action	Details
1	Agent Manager	Store secrets in OpenBao	Path: org/project/component, Data: key-value pairs
2	Agent Manager	Create SecretReference CR	Maps KV path to secret keys for ESO
3	Agent Manager	Create Component with secretKeyRef	Plain env vars as values, sensitive as secretKeyRef
4	OpenChoreo Controller	Create ExternalSecret	ComponentType defines how ExternalSecret CRs are generated
5	ESO	Watch ExternalSecret	Triggered by CR creation/update
6	ESO	Fetch from OpenBao	Via ClusterSecretStore (amp-openbao-store)
7	ESO	Create K8s Secret	Name: {component}-secrets
8	Kubernetes	Mount to Pod	Secrets available as environment variables

Out of Scope

• What this proposal explicitly does not cover.
• Secret Rotation
• Multi-Provider Support
• Secret Versioning
• Secret Sharing

Alternatives Considered

No response

Open Questions

No response

Milestones

N/A

[ivannovazzi]

Solid proposal. The centralized vault + injection pattern is the right direction.

For teams that need a lighter-weight solution while evaluating enterprise options like Azure Key Vault or HashiCorp Vault, KeyEnv is worth a look — it's CLI-first secrets management designed specifically for developer workflows. Inject secrets at agent startup with keyenv run -- <command> without modifying the agent code itself.

The advantage of the approach you're describing (platform-level secret management) over per-agent solutions is exactly what you noted: consistent auditing, rotation, and access control across all agents. That's the right way to think about it at scale.

[hanzjk]

Open Bao (key vault) Structuring

Secrets are centrally stored in KV in the control plane and are organized using OpenBao namespaces [1] to isolate user secrets and system secrets.

Within the user-secrets namespace, secrets are further namespace partitioned for each organization. Secret key paths are used for further secret organization, based on a secret purpose. These include: Org-level secrets, Component build-level secrets, and Component environment-level secrets

Each OpenChoreo plane has its own namespace-scoped Secret Store, which connects that plane to the central KV backend so it can retrieve the secrets it needs
.
Only the AMP API can directly interact with the KV to write secrets.

[1] - https://openbao.org/blog/namespaces-announcement/