[Proposal] Introduce RuntimeDeployConfig as Kind-Agnostic IR in Gateway Controller

[renuka-fernando]

Summary

• Introduce RuntimeDeployConfig as a new kind-agnostic intermediate representation (IR) in the gateway controller, replacing the current pattern where all deployment kinds (RestAPI, LLM Provider, LLM Proxy, MCP) are internally converted to RestAPI
• Each deployment kind gets its own ConfigTransformer implementation that produces RuntimeDeployConfig from stored configuration
• The Envoy xDS translator and policy xDS translator consume RuntimeDeployConfig exclusively, decoupled from any specific API kind
• Introduce PolicyChainResolver as a named, registered component in the Policy Engine that determines which policy chain applies to a given request, enabling kind-specific resolution strategies (e.g., route-key based vs. JSON-RPC body parsing)
• Deliver API metadata directly to the Policy Engine via xDS at deployment time instead of injecting it into Envoy route dynamic metadata, eliminating expensive prototext.Unmarshal parsing at request time (performance improvement)
• Add a second xDS cache in the Policy Engine for RouteConfig resources alongside the existing PolicyChain cache

Proposal

• Define RuntimeDeployConfig as the single IR struct that all xDS translators consume
• Implement the ConfigTransformer interface per kind: RestAPITransformer, LLMTransformer, MCPTransformer
• Replace all direct RestAPI usage in CreateRoute, CreateCluster, and policy snapshot generation with RuntimeDeployConfig
• Introduce PolicyChainResolver as a pluggable component in the Policy Engine, with route-key and mcp-tool as the initial implementations
• Split the Policy Engine's xDS data into two caches: RouteConfig (metadata + resolver name) and PolicyChain (existing)

Key Design Decisions

RuntimeDeployConfig as the central IR

type RuntimeDeployConfigs struct {
    Configs map[string]*RuntimeDeployConfig
}

type RuntimeDeployConfig struct {
    Metadata            Metadata
    Context             string              // API base path (e.g. "/weather/$version"); "" for kinds with no context (LLM Proxy)
    PolicyChainResolver string              // name of resolver registered in PE (e.g. "route-key", "mcp-tool")
    Routes              map[string]*Route
    PolicyChains        map[string]*PolicyChain
    UpstreamClusters    map[string]*UpstreamCluster
}

type Metadata struct {
    Kind        string
    Handle      string
    Name        string
    Version     string
    DisplayName string
    ProjectID   string
    LLM         *LLMMetadata // nil for non-LLM kinds
}

type LLMMetadata struct {
    TemplateHandle string
    ProviderName   string
}

type Route struct {
    Method          string
    Path            string // full path including context prefix (set by transformer)
    Vhost           string // "" = default vhost
    AutoHostRewrite bool
    Timeout         RouteTimeout
    Upstream        RouteUpstream
}

type RouteTimeout struct {
    Connect *time.Duration
}

type RouteUpstream struct {
    DefaultUpstreamClusterKey string // links to UpstreamClusters map
    UseClusterHeader          bool   // if true, policy selects upstream dynamically
}

type PolicyChain struct {
    Policies []Policy
}

type Policy struct {
    Name               string
    Version            string
    Params             map[string]interface{}
    ExecutionCondition string
}

type UpstreamCluster struct {
    BasePath  string
    Endpoints []Endpoint
}

// Use Host + Port, not a full URL — Envoy clusters only need
// host (for DNS resolution) and port (for socket address).
type Endpoint struct {
    Host string // e.g. "backend.example.com"
    Port int    // e.g. 8443
    TLS  *UpstreamTLS
}

type UpstreamTLS struct {
    Enabled bool
    Certs   map[string][]byte
}

The struct captures everything the xDS translators need to configure Envoy and the Policy Engine without knowing which kind produced the configuration. The Context field is empty for kinds like LLM Proxy that have no base path concept. The PolicyChainResolver field declares which resolver the Policy Engine should use for this deployment.

PolicyChainResolver in the Policy Engine

// PolicyChainResolver is a named, registered component in the PE that resolves
// which PolicyChain to apply for a given request. Like a policy, it declares
// what request data it needs before it can run.
type PolicyChainResolver interface {
    Name() string
    Requirements() ResolverRequirements
    Resolve(ctx ResolverContext) (policyChainKey string, err error)
}

type ResolverRequirements struct {
    BufferBody bool // true = whole request body must be buffered before Resolve() is called; streaming not supported for chain resolution
    Headers    bool
}

⚠️ Important: Streaming is not supported for chain resolution. If BufferBody is true, the entire request body must be buffered before the resolver can run.

The resolver is a named, registered component that determines which policy chain to apply for a given request. Like a policy, it declares what request data it needs before it can run:

Resolver	BufferBody	Logic	Used By
route-key	false	policyChainKey = routeKey	RestAPI, LLM Provider, LLM Proxy
mcp-tool	true	Parse JSON-RPC body, extract tool name	MCP

This directly impacts the ext_proc flow:

Request headers arrive
  -> get routeKey from Envoy dynamic metadata
  -> look up resolver for this route config
  -> Does resolver need body?
     YES (mcp-tool): buffer body -> resolver.Resolve(body) -> policyChainKey
     NO  (route-key): resolver.Resolve(routeKey) -> policyChainKey immediately
  -> look up PolicyChain[policyChainKey]
  -> execute policies

Two xDS caches in the Policy Engine

// Kernel holds two separate maps, each backed by its own xDS cache.
type Kernel struct {
    RouteConfigs map[string]*RouteConfig         // routeKey → config (xDS cache 1)
    PolicyChains map[string]*registry.PolicyChain // policyChainKey → chain (xDS cache 2)
}

// RouteConfig does NOT hold PolicyChains — it only references the resolver by name.
// Metadata is pre-populated at deploy time; no request-time parsing needed.
type RouteConfig struct {
    Metadata                 Metadata              // API metadata — set at deploy time, no request-time parsing
    ResolverName             string                // name of PolicyChainResolver registered in PE
    // NOTE: These fields don't conceptually belong to a route — they belong to
    // the deployment artifact (RestAPI, LLM Provider, etc.). Placed here because
    // there is no dedicated xDS resource for deployment artifacts. Improve later
    // by introducing a separate xDS cache for deployment-level config.
    UpstreamBasePath         string                // base path of the default upstream (for path context in policies)
    UpstreamDefinitionPaths  map[string]string     // upstream name → base path (for dynamic path rewriting when UpstreamName policy selects a different upstream)
}

xDS Cache	Key	Value	Source in RuntimeDeployConfig
RouteConfigs	routeKey	Metadata + ResolverName + UpstreamBasePath	Routes map + Metadata + PolicyChainResolver
PolicyChains	policyChainKey	Ordered list of policies	PolicyChains map

The split means that at request time, the Policy Engine receives only the routeKey from Envoy. It looks up the RouteConfig to get pre-populated metadata (no parsing) and the resolver name. The resolver then determines the policy chain key, and the chain is looked up from the second cache.

Metadata delivery via xDS (not Envoy route metadata)

Currently, createRoute injects dynamic metadata (api_id, api_name, api_version, api_kind, etc.) into Envoy routes. At request time, Envoy forwards this to the Policy Engine via ext_proc, which runs extractRouteMetadata() involving prototext.Unmarshal (expensive, ~11% CPU, required a cache).

With this proposal, metadata is sent directly to the Policy Engine via the RouteConfig xDS resource at deployment time. It is stored in memory and looked up by routeKey at request time. No parsing, no cache needed.

PolicyChain resolution by kind

Kind	Resolver	PolicyChain Key Convention
RestAPI	route-key	policyChainKey = routeKey (e.g. GET_petstore_v1_/pets_{id})
LLM Provider	route-key	Same as RestAPI
LLM Proxy	route-key	Same as RestAPI
MCP	mcp-tool	Tool name or tool group extracted from JSON-RPC body

Drawbacks

• Migration scope: The xDS translator (translator.go) is one of the largest files in the gateway controller. Refactoring it to consume RuntimeDeployConfig instead of RestAPI is a significant change that touches critical code paths.
• Two xDS caches add complexity: The Policy Engine currently has a single policy xDS cache. Adding a second RouteConfig cache introduces additional snapshot management, versioning, and debugging surface area.
• Body buffering for MCP: The mcp-tool resolver requires full request body buffering before chain resolution, which adds latency for MCP requests. This is inherent to the JSON-RPC protocol and unavoidable, but it is a tradeoff that does not exist for other kinds.
• Incremental rollout risk: Until all kinds are fully migrated to their own transformers, there may be a period where some code paths still use RestAPI while others use RuntimeDeployConfig, creating two parallel paths to maintain.