[Proposal] Rewrite Path and Dynamic Endpoint - Resolve Conflicting Policies

[renuka-fernando]

Summary

• Problem: Rewrite Path requires clear_route_cache: false (to avoid re-matching against the new path), while Dynamic Endpoint requires clear_route_cache: true (to re-evaluate the cluster from the cluster_header). Both cannot coexist in a single ext proc filter.
• Solution: Split responsibilities across two filters — ext proc sets the cluster header and clears the route cache, then a Lua filter (with clear_route_cache: false) modifies :path/:method without triggering re-evaluation.
• Route config: Routes must use cluster_header: x-wso2-target-upstream-cluster and request_headers_to_remove to strip the internal header before forwarding upstream.
• Why it works: Lua's decodeHeaders() triggers lazy route re-evaluation before the Lua script runs — at that point the original :path is intact and the new cluster header is present.
• Drawbacks:
  • Route matching runs twice per request (initial + re-evaluation after cache clear), adding CPU overhead.
  • Ext proc server must always process request headers to set the cluster header — cannot skip processing even when no policies apply.
  • Both filters execute on every request regardless of whether policies are configured.

Demo Docker Compose Setup with Envoy + Ext Proc + Lua filters: envoy-external-processor-sample.zip

1. Problem

In the current implementation of the policy engine, both the Rewrite Path action and the Dynamic Endpoint action require different settings for clear_route_cache in Envoy, which leads to a conflict when both actions are present in the same policy.

1.1. Rewrite Path Action

The Rewrite Path action allows you to modify the path of incoming requests before they are forwarded to the upstream service. To accomplish this, we can change the header :path and we should not do clear_route_cache on Envoy, otherwise the route cache will be cleared and the new path Envoy will reevaluate the route and may cause unexpected routing results.

Note: The same applies to :method (method change) and :authority (host change) — modifying these headers with clear_route_cache enabled will also trigger route re-evaluation, which may cause unexpected routing results.

Example:

routes:
  - match:
      prefix: /api-1
    route:
      cluster_header: x-wso2-target-upstream-cluster
      timeout: 10s # 10 seconds
    request_headers_to_remove:
      - x-wso2-target-upstream-cluster
  - match:
      prefix: /api-2
    route:
      cluster_header: x-wso2-target-upstream-cluster
      timeout: 60s # 60 seconds
    request_headers_to_remove:
      - x-wso2-target-upstream-cluster

If a policy rewrites the path from /api-1 to /api-2, the request will be routed to the second route and the timeout will be 60 seconds. If the policy rewrites the path from /api-1 to /foo, the request will not match any route and responds with 404 Not Found.

Hence clear_route_cache should be set to false to avoid unexpected routing results.

1.2. Dynamic Endpoint Action

The Dynamic Endpoint action allows you to modify the upstream cluster of incoming requests so that they can be forwarded to different upstream services. We can use cluster_header to specify the header that contains the name of the target upstream cluster. But in this case we have to set clear_route_cache to true to clear out the already selected cluster and force Envoy to re-evaluate the route and select the new cluster specified in the header.

So we can't achieve both actions at the same time because they have conflicting requirements for clear_route_cache.

2. What clear_route_cache Does

When a request arrives, Envoy resolves the route and caches both the matched route and cluster info. The clear_route_cache operation wipes this cached state. The re-evaluation does not happen immediately at the point of clearing — it happens lazily when the next filter in the chain accesses the route (typically the router filter, or any intermediate filter that calls route()).

The re-evaluation is a full route matching from scratch: virtual host matching, route matching (prefix/path/regex/headers), and cluster resolution (including reading cluster_header). All route-level settings (timeouts, retry policies, buffer limits, etc.) are also refreshed from the newly matched route.

Check section 6.1 below for more details.

3. Solution

This solution requires cluster_header (x-wso2-target-upstream-cluster) to be configured in the route section, which tells Envoy to read the target cluster name from a request header instead of a static cluster assignment. With this in place, an ext proc filter followed by a Lua filter is used to handle both actions. Each filter is responsible for the mutations that match its clear_route_cache requirement:

• Ext Proc Filter: Updates the cluster header and sets clear_route_cache to true on every request. This is required because cluster_header is evaluated during route matching — before any HTTP filter runs, so the initial route always has an empty cluster name. The ext proc filter must always set the cluster header and clear the route cache for the cluster to resolve correctly.
• Lua Filter: Updates :path and/or :method headers without clearing the route cache (clear_route_cache: false). This is necessary because clearing the route cache after changing :path would cause Envoy to re-match the route against the new path, which may not match any route (404) or match a different route with different settings (wrong timeouts, retries, etc.).

The cluster header (x-wso2-target-upstream-cluster) is an internal routing header and should not be forwarded to the upstream service. Use request_headers_to_remove in the route configuration to strip it before the request is sent upstream.

The ext proc filter communicates the desired path/method changes to the Lua filter via dynamic metadata, so the Lua filter applies them locally without a gRPC round-trip.

http_filters:
  - name: envoy.filters.http.ext_proc
    typed_config:
      "@type": type.googleapis.com/envoy.extensions.filters.http.ext_proc.v3.ExternalProcessor
      grpc_service:
        envoy_grpc:
          cluster_name: ext-proc-server
      processing_mode:
        request_header_mode: SEND

  - name: envoy.filters.http.lua
    typed_config:
      "@type": type.googleapis.com/envoy.extensions.filters.http.lua.v3.Lua
      clear_route_cache:
        value: false
      inline_code: |
        function envoy_on_request(request_handle)
          local metadata = request_handle:streamInfo():dynamicMetadata()
          local ns = metadata:get("policy.engine")
          if not ns then
            return
          end
          if ns["rewrite_path"] then
            request_handle:headers():replace(":path", ns["rewrite_path"])
          end
          if ns["change_method"] then
            request_handle:headers():replace(":method", ns["change_method"])
          end
        end

  - name: envoy.filters.http.router

3.1. Ext Proc Filter

• Executes the policy engine logic.
• Always sets the cluster header (x-wso2-target-upstream-cluster) with the target cluster name and sets clear_route_cache to true. This is required because cluster_header is evaluated during initial route matching — before any HTTP filter runs. At that point the header is not present, so the initial cached route has an empty cluster name. The ext proc filter must always set this header and clear the route cache to trigger re-evaluation with the correct cluster.
• If there is a rewrite path or change HTTP method action, it sets dynamic metadata (e.g., rewrite_path or change_method) to communicate the desired changes to the Lua filter. It does not modify :path or :method directly — that is delegated to the Lua filter to avoid the conflict with clear_route_cache.

3.2. Lua Filter

• Configured with clear_route_cache: false to prevent route cache clearing on header modifications. This is critical — without this, the default Lua behavior would clear the route cache on every header modification, causing the new :path to be used in route re-matching.
• Reads dynamic metadata set by the ext proc filter.
• If rewrite path or change HTTP method metadata is present, it updates the :path or :method header accordingly.
• Since clear_route_cache is disabled, modifying headers does not trigger route re-evaluation, preserving the original route and all its settings (timeouts, retries, rate limits, etc.).

3.3. Why This Works

The key insight is the lazy route re-evaluation behavior and the ordering of operations across the two filters:

1. Ext proc sets the cluster header and clears the route cache. The cache is wiped but no re-evaluation happens yet — clearRouteCache() only sets cached_route_ to an empty optional.

2. The Lua filter's decodeHeaders() triggers route re-evaluation. At the start of decodeHeaders(), the Lua filter calls getPerLuaCodeSetup() → resolveMostSpecificPerFilterConfig() → getRoute() → route(). Since the cache is empty, this triggers refreshCachedRoute() — a full route re-match. At this point, the original :path is still intact (the Lua script hasn't run yet) and the new cluster header is present (ext proc already set it). The route matches correctly with the proper cluster and all original route settings.

3. The Lua script runs and modifies :path/:method. Since clear_route_cache is false, these header modifications do not clear the route cache. The correctly-resolved cached route from step 2 persists.

4. The router filter uses the cached route. It sees a valid cached route with the correct cluster, original timeouts, and all other route settings intact. The modified :path is forwarded to the upstream as-is.

sequenceDiagram
    participant EP as Ext Proc Filter
    participant LUA as Lua Filter<br/>(decodeHeaders)
    participant LS as Lua Script
    participant R as Router Filter

    Note over EP: cached_route_ = [old route]<br/>:path = "/api-1"<br/>cluster header = (empty)

    EP->>EP: Set cluster header = "my-svc"
    EP->>EP: clear_route_cache: true
    Note over EP: cached_route_ = [empty]<br/>:path = "/api-1"<br/>cluster header = "my-svc"

    EP->>LUA: Continue to next filter

    LUA->>LUA: getPerLuaCodeSetup()<br/>→ resolveMostSpecificPerFilterConfig()<br/>→ route() → cache empty<br/>→ refreshCachedRoute()
    Note over LUA: Route re-evaluated with<br/>original :path "/api-1" +<br/>new cluster header "my-svc"<br/>cached_route_ = [new route ✓]

    LUA->>LS: Run Lua script

    LS->>LS: Replace :path = "/new-path"<br/>(clear_route_cache: false → no cache clear)
    Note over LS: cached_route_ = [new route ✓]<br/>:path = "/new-path"<br/>cluster header = "my-svc"

    LS->>R: Continue to next filter

    R->>R: callbacks_->route()<br/>→ cache valid → use cached route
    Note over R: Routes to cluster "my-svc"<br/>with original route settings<br/>(timeouts, retries, etc.)<br/>:path "/new-path" forwarded upstream

Loading

Check section 6.2 below for more details.

3.4. Why a Single Ext Proc Filter Cannot Achieve Both

Within a single ext proc HeadersResponse, all header mutations are applied first, then clearRouteCache is called after (processor_state.cc#L175-L185):

// Step 1: ALL header mutations applied (both cluster header + :path)
processHeaderMutation(common_response, ...);

// Step 2: THEN route cache cleared
clearRouteCache(common_response);

There is no way to split these within a single ext proc response. So if you set both the cluster header and change :path in the same response with clear_route_cache: true, the route re-evaluation will see the new :path, which may not match any route (404) or match a different route with different settings.

4. Drawbacks

• The ext proc filter and Lua filter are always executed for every request, whether there is a policy which will do dynamic endpoint actions or not or even there are no policies at all, which may introduce some performance overhead. However, the Lua filter overhead is minimal compared to a second ext proc gRPC call.
• Clearing the route cache on every request forces Envoy to perform route matching twice per request: once on initial request arrival, and again when the next filter (Lua) accesses the route after the cache is cleared. The re-evaluation involves virtual host lookup (findVirtualHost), route entry matching (prefix/path/regex/header matching via getRouteFromEntries), cluster specifier plugin invocation (clusterEntry), thread-local cluster info lookup (getThreadLocalCluster), and refreshing timeouts, idle/flush timeouts, buffer limits, and tracing. While these are all in-memory operations with no I/O, they do add CPU overhead on the request hot path for every request.
• The ext proc server must always process request headers and set the x-wso2-target-upstream-cluster header with the upstream cluster name on every request. Previously, the ext proc server could skip processing when no policies were applicable. Now the ext proc request header processing cannot be bypassed, as the route will have an empty cluster name without it.

5. Alternatives

5.1. Remove Dynamic Endpoint Support from the Gateway at all

So only the existing ext proc filter is required.

Rejected: Dynamic Endpoint support is required.

5.2. Lua Cluster Specifier Plugin

The route matching is performed at the time the request headers are received. The same clearRouteCache requirement applies.

- name: upstream-1
    match:
    prefix: "/svc-1"
    route:
    inline_cluster_specifier_plugin:
        extension:
        name: envoy.router.cluster_specifier_plugin.lua
        typed_config:
            "@type": type.googleapis.com/envoy.extensions.router.cluster_specifiers.lua.v3.LuaConfig
            source_code:
            inline_string: |
                function envoy_on_route(route_handle)
                local cluster = route_handle:headers():get("x-target-cluster")
                route_handle:logInfo("Lua cluster specifier: x-target-cluster=" .. (cluster or "nil"))
                if cluster then
                    return cluster
                end
                return "upstream-service-1"
                end
            default_cluster: upstream-service-1

Rejected: This won't work because the Lua cluster specifier plugin (and inline_cluster_specifier_plugin in general) is evaluated during route matching via the same clusterEntry() code path as cluster_header. All cluster specifier plugins (cluster_header, inline_cluster_specifier_plugin, cluster_specifier_plugin, weighted_clusters) are assigned to the same cluster_specifier_plugin_ field on the route and invoked at route resolution time. The ext proc filter hasn't run yet when route matching happens, so the header value is not present. Check section 6.3 below for more details.

5.3. Change Cluster Directly - Write CPP Extension

Write a cpp extension to directly set the cluster based on a dynamic metadata set from the ext proc filter, which will not require to set clear_route_cache to true and avoid the conflict between rewrite path and dynamic endpoint actions.

This would use the setRoute() API available on filter callbacks to wrap the current route in a DynamicRouteEntry that only overrides clusterName(), preserving all other route settings (timeouts, retries, rate limits, etc.) without any route re-matching.

Check section 6.4 below for more details.

Rejected: This approach requires writing a custom CPP extension, which adds complexity and maintenance overhead.

6. Envoy Code Analysis - Detailed Explanation

6.1. Clear Route Cache and Reevaluate Route - Envoy Code Analysis

6.1.1. clearRouteCache() — Wipe the cached state

conn_manager_impl.cc#L2411-L2420:

void ConnectionManagerImpl::ActiveStream::clearRouteCache() {
  if (routeCacheBlocked()) {
    return;
  }
  setCachedRoute({});            // cached_route_ = empty optional (has_value() == false)
  cached_cluster_info_ = {};     // cached_cluster_info_ = empty optional
}

This sets cached_route_ to an empty optional (has_value() == false). No route matching happens here — it only wipes the cache.

6.1.2. Lazy re-resolution when route() is called

conn_manager_impl.cc#L2287-L2294:

Router::RouteConstSharedPtr
ConnectionManagerImpl::ActiveStream::route(const Router::RouteCallback& cb) {
  if (cached_route_.has_value()) {   // after clearRouteCache(), this is FALSE
    return cached_route_.value();
  }
  refreshCachedRoute(cb);            // full re-resolution triggered here
  return cached_route_.value();
}

Since clearRouteCache() set cached_route_ to an empty optional, has_value() returns false, triggering refreshCachedRoute(). The same lazy pattern applies to clusterInfo() at conn_manager_impl.cc#L2278-L2286.

6.1.3. refreshCachedRoute() — Full route matching from scratch

conn_manager_impl.cc#L1778-L1800:

void ConnectionManagerImpl::ActiveStream::refreshCachedRoute(const Router::RouteCallback& cb) {
  if (routeCacheBlocked()) { return; }

  Router::VirtualHostRoute route_result;
  if (request_headers_ != nullptr) {
    // ...scoped route handling...
    if (snapped_route_config_ != nullptr) {
      route_result = snapped_route_config_->route(cb, *request_headers_,   // full route match
                                                  filter_manager_.streamInfo(), stream_id_);
    }
  }
  setVirtualHostRoute(std::move(route_result));  // cache new result
}

snapped_route_config_->route() runs the entire route matching pipeline again against the current state of the request headers.

6.1.4. cluster_header is read during route matching

During route matching, once the path/headers match a route entry, clusterEntry() is called — config_impl.cc#L1320-L1327:

RouteConstSharedPtr RouteEntryImplBase::clusterEntry(const Http::RequestHeaderMap& headers, ...) {
  if (cluster_specifier_plugin_ != nullptr) {
    return cluster_specifier_plugin_->route(shared_from_this(), headers, ...);
  }
  return shared_from_this();
}

For routes configured with cluster_header, the plugin is HeaderClusterSpecifierPlugin — header_cluster_specifier.cc#L11-L24:

RouteConstSharedPtr HeaderClusterSpecifierPlugin::route(..., const Http::RequestHeaderMap& headers, ...) {
  const auto entry = headers.get(cluster_header_);    // reads the header NOW
  absl::string_view cluster_name;
  if (!entry.empty()) {
    cluster_name = entry[0]->value().getStringView();  // gets the value
  }
  return std::make_shared<DynamicRouteEntry>(std::move(parent), std::string(cluster_name));
}

The cluster name is read from the header and baked into a DynamicRouteEntry as a const std::string — delegating_route_impl.h#L143-L152:

class DynamicRouteEntry : public DelegatingRouteEntry {
public:
  DynamicRouteEntry(RouteConstSharedPtr route, std::string&& cluster_name)
      : DelegatingRouteEntry(std::move(route)), cluster_name_(std::move(cluster_name)) {}

  const std::string& clusterName() const override { return cluster_name_; }

private:
  const std::string cluster_name_;  // immutable, set once at route matching time
};

6.1.5. setVirtualHostRoute() — Cache the new route + cluster info

conn_manager_impl.cc#L2313-L2344:

void ConnectionManagerImpl::ActiveStream::setVirtualHostRoute(Router::VirtualHostRoute vhost_route) {
  Router::RouteConstSharedPtr route = std::move(vhost_route.route);
  setCachedRoute({route});                                          // cache new route

  if (nullptr == route || nullptr == route->routeEntry()) {
    cached_cluster_info_ = nullptr;
  } else {
    auto* cluster = connection_manager_.cluster_manager_.getThreadLocalCluster(
        route->routeEntry()->clusterName());                        // lookup cluster by new name
    cached_cluster_info_ = (nullptr == cluster) ? nullptr : cluster->info();
  }

  filter_manager_.streamInfo().route_ = std::move(route);
  filter_manager_.streamInfo().setUpstreamClusterInfo(cached_cluster_info_.value());

  refreshDurationTimeout();      // recalculate timeouts from new route
  refreshIdleAndFlushTimeouts(); // recalculate idle timeouts
  refreshBufferLimit();          // recalculate buffer limits
}

6.1.6. What gets re-evaluated

What	Re-evaluated?	How
Virtual host	Yes	snapped_route_config_->route() re-matches vhost
Route (path/prefix/header matching)	Yes	Full route matching runs again
Cluster name (via cluster_header)	Yes	HeaderClusterSpecifierPlugin::route() re-reads the header
Cluster info	Yes	getThreadLocalCluster(clusterName()) looks up the new cluster
Timeouts	Yes	refreshDurationTimeout(), refreshIdleAndFlushTimeouts()
Buffer limits	Yes	refreshBufferLimit()

6.2. Route Re-evaluation in decodeHeaders() Stage - Envoy Code Analysis

The route re-evaluation after clearRouteCache() does not happen immediately. It is triggered lazily when the next filter in the chain accesses the route during its decodeHeaders() stage. This is a common pattern across most HTTP filters — not specific to ext proc or Lua.

Most HTTP filters that support per-route configuration access the route at the very start of decodeHeaders() to resolve per-route config overrides. This includes ext proc (mergePerRouteConfig()), Lua (getPerLuaCodeSetup()) and others. This route access triggers lazy re-evaluation if the route cache was previously cleared. Simple filters like health_check, ip_tagging, adaptive_concurrency, and grpc_stats do not access the route during decodeHeaders() and therefore would not trigger re-evaluation.

Regardless of which filter triggers it, the common code path is through the filter manager's getRoute() — filter_manager.cc#L300-L305:

Router::RouteConstSharedPtr ActiveStreamFilterBase::getRoute() const {
  if (parent_.filter_manager_callbacks_.downstreamCallbacks()) {
    return parent_.filter_manager_callbacks_.downstreamCallbacks()->route(nullptr);
  }
  return parent_.streamInfo().route();
}

This calls ActiveStream::route() which triggers refreshCachedRoute() if the cache was cleared (as shown in section 6.1.2 above).

This is the key mechanism that makes the ext proc + Lua solution work: the Lua filter's decodeHeaders() accesses the route (via getPerLuaCodeSetup() → resolveMostSpecificPerFilterConfig() → getRoute()) before the Lua script runs. At that point, the original :path is still intact and the new cluster header is present, so the route re-evaluates correctly. Then the Lua script modifies :path without clearing the cache, preserving the correctly-resolved route.

If no intermediate filter accesses the route, the router filter's decodeHeaders() will trigger re-evaluation as a last resort — router.cc#L471:

route_ = callbacks_->route();  // triggers refreshCachedRoute() if cache was cleared
if (!route_) {
  stats_.no_route_.inc();
  callbacks_->sendLocalReply(Http::Code::NotFound, ...);
  return Http::FilterHeadersStatus::StopIteration;
}

6.3. Lua Cluster Specifier Plugin - Envoy Code Analysis

All cluster specifier plugins (cluster_header, inline_cluster_specifier_plugin, cluster_specifier_plugin, weighted_clusters) are assigned to the same cluster_specifier_plugin_ field on the route entry and share the same code path — config_impl.cc#L609-L627:

if (route.route().has_weighted_clusters()) {
    cluster_specifier_plugin_ = std::make_shared<WeightedClusterSpecifierPlugin>(...);
} else if (route.route().has_inline_cluster_specifier_plugin()) {
    cluster_specifier_plugin_ = getClusterSpecifierPluginByTheProto(
        route.route().inline_cluster_specifier_plugin(), ...);
} else if (route.route().has_cluster_specifier_plugin()) {
    cluster_specifier_plugin_ = vhost_->globalRouteConfig().clusterSpecifierPlugin(
        route.route().cluster_specifier_plugin());
} else if (route.route().has_cluster_header()) {
    cluster_specifier_plugin_ =
        std::make_shared<HeaderClusterSpecifierPlugin>(route.route().cluster_header());
}

All of these are called through the same clusterEntry() at route matching time — config_impl.cc#L1320-L1327:

RouteConstSharedPtr RouteEntryImplBase::clusterEntry(const Http::RequestHeaderMap& headers, ...) {
  if (cluster_specifier_plugin_ != nullptr) {
    return cluster_specifier_plugin_->route(shared_from_this(), headers, ...);
  }
  return shared_from_this();
}

The plugin's route() method receives the request headers at route matching time. Since route matching occurs before any HTTP filter runs, any header value that an ext proc filter would set is not yet present. Using inline_cluster_specifier_plugin with a Lua cluster specifier (or any other plugin) does not change this fundamental timing issue — the same clearRouteCache requirement applies.

6.4. setRoute() — Directly Set a Route with the Desired Cluster

The setRoute() API is available on filter callbacks — filter.h#L350:

virtual void setRoute(Router::RouteConstSharedPtr route) PURE;

A custom C++ filter could get the current route and wrap it in a DynamicRouteEntry with the desired cluster name:

auto current_route = callbacks_->route();
auto new_route = std::make_shared<Router::DynamicRouteEntry>(
    current_route, std::string("my-target-cluster"));
callbacks_->setRoute(new_route);

DynamicRouteEntry delegates everything to the original route except clusterName() — delegating_route_impl.h#L143-L152:

class DynamicRouteEntry : public DelegatingRouteEntry {
public:
  DynamicRouteEntry(RouteConstSharedPtr route, std::string&& cluster_name)
      : DelegatingRouteEntry(std::move(route)), cluster_name_(std::move(cluster_name)) {}
  const std::string& clusterName() const override { return cluster_name_; }
private:
  const std::string cluster_name_;
};

This replaces the cached route directly — no route re-matching at all. All original route settings (timeouts, retry policies, rate limits, CORS, hash policies, etc.) are preserved. A route carries much more than just a cluster name — the RouteEntry interface defines over 30 route-level settings (router.h#L916-L1175).

However, no existing HTTP filter (ext proc, Lua, or any other) calls setRoute() today. Using this approach would require writing a custom C++ HTTP filter.

There is also refreshRouteCluster() available as a lighter-weight alternative to clearRouteCache() — filter.h#L386. It refreshes only the cluster without re-matching the route. However, cluster_header's DynamicRouteEntry stores the cluster name as const std::string and the base RouteEntryImplBase::refreshRouteCluster() is a no-op (config_impl.h#L591-L592), so this approach would not work with cluster_header without modifying Envoy core.

[renuka-fernando]

Safeguard: Prevent Ext Proc from Modifying System Headers

Since the ext proc filter is only responsible for setting the x-wso2-target-upstream-cluster cluster header and clearing the route cache, we can add mutation_rules with disallow_system: true to the ext proc filter config as an additional safeguard. This prevents the ext proc server from modifying system headers (:host, :authority, :scheme, :method, :path) — even if the policy engine or ext proc server has a bug that attempts to mutate them.

- name: envoy.filters.http.ext_proc
  typed_config:
    "@type": type.googleapis.com/envoy.extensions.filters.http.ext_proc.v3.ExternalProcessor
    grpc_service:
      envoy_grpc:
        cluster_name: ext-proc-server
    processing_mode:
      request_header_mode: SEND
    mutation_rules:
      disallow_system: true
      disallow_is_error: true

With disallow_is_error: true, Envoy will reject the entire ext proc response and return an error to the client when a disallowed system header mutation is attempted, rather than silently ignoring it. This ensures invalid mutations are caught immediately instead of being masked.

Additionally, the policy engine should validate if a policy is trying to update system headers like host, :authority, :scheme, :method, or :path directly from the ext proc server and log an ERROR indicating that these mutations should be handled by the Lua filter instead. Since disallow_is_error: true is set, Envoy will reject the ext proc response and return a 500 Internal Server Error to the client.

[renuka-fernando]

Problem Explanation with Example

Scenario 1: Without Dynamic Endpoint — No Issues

Upstream: http://httpbin.org/anything

Routes:
- GET /abc:
  - timeout = 50s
  - if headers[env] == "dev" → path = /abc/foo
  - if headers[env] == "prod" → path = /abc/bar
  - substitutePath /abc → /anything
- GET /abc/foo:
  - timeout = 5s
  - substitutePath /abc/foo → /anything
- GET /abc/bar:
  - timeout = 20s
  - substitutePath /abc/bar → /anything

A request GET /abc with env: dev header arrives:

1. Envoy matches route GET /abc (timeout = 50s).
2. Ext proc evaluates the policy — rewrites :path from /abc to /abc/foo (conditional path rewrite) and applies substitutePath to get /anything.
3. No clear_route_cache needed — the original matched route (GET /abc, timeout = 50s) is preserved.
4. Request is forwarded to httpbin upstream with :path = /anything and timeout = 50s.

Everything works as expected. The rewritten path is only used for forwarding to upstream — it does not affect route matching.

Scenario 2: With Dynamic Endpoint — The Conflict

Now a Dynamic Endpoint action is added to change the upstream cluster:

Upstreams:
  - name: httpbin
    url: http://httpbin.org/anything
  - name: httpbin2
    url: http://httpbin.org/anything2

Routes:
- GET /abc:
  - timeout = 50s
  - if headers[env] == "dev" → path = /abc/foo
  - if headers[env] == "prod" → path = /abc/bar
  - substitutePath /abc → /anything
  - upstream = httpbin2 ### SETTING a Dynamic Endpoint here ..........
- GET /abc/foo:
  - timeout = 5s
  - substitutePath /abc/foo → /anything
- GET /abc/bar:
  - timeout = 20s
  - substitutePath /abc/bar → /anything

A request GET /abc with env: dev header arrives. Using a single ext proc filter, the ext proc server responds with:

• Set x-wso2-target-upstream-cluster: httpbin2 (dynamic endpoint)
• Rewrite :path from /abc to /abc/foo (conditional path rewrite)
• clear_route_cache: true (required for the cluster change to take effect)

What goes wrong

Within ext proc's handleHeadersResponse, all header mutations are applied first, then clear_route_cache is called after. So at the point of route re-evaluation:

• :path is already /abc/foo (not the original /abc)
• x-wso2-target-upstream-cluster is httpbin2

Envoy re-matches the route against :path = /abc/foo and matches GET /abc/foo instead of the original GET /abc. This causes:

Setting	Expected (from GET /abc)	Actual (from GET /abc/foo)
Timeout	50s	5s
substitutePath	/abc → /anything	/abc/foo → /anything
Other route settings	From GET /abc route	From GET /abc/foo route

The request ends up with the wrong timeout (5s instead of 50s) and potentially wrong route-level settings. If the rewritten path doesn't match any route at all, the request gets a 404 Not Found.

[renuka-fernando]

Update upstreamDefinitions to support dynamic EP.
#369 (comment)