Deployment Acknowledgement via WebSocket

[malinthaprasan]

Deployment Acknowledgement via WebSocket

Summary

Add deployment acknowledgement via WebSocket so the control plane accurately tracks whether a deployment actually succeeded on the gateway, rather than optimistically marking it as deployed.

Problem

Today, when the control plane pushes a deployment event (API, LLM Provider, or LLM Proxy) to a gateway via WebSocket, it immediately sets the status to DEPLOYED — before the gateway has actually processed it. There is no feedback loop:

• If the gateway fails to process a deployment, the control plane still shows it as DEPLOYED
• There is no way to know if a deployment is currently being processed by the gateway
• Failed deployments are invisible to users and operators
• On gateway disconnect during deployment processing, the state is inconsistent

This affects all WebSocket-based deployment types: REST APIs, LLM Providers, and LLM Proxies.

Proposed Solution

New Deployment States

Introduce intermediate and terminal states for the deployment lifecycle:

DEPLOYING → DEPLOYED (gateway confirms success)
DEPLOYING → FAILED (gateway reports failure or timeout)
UNDEPLOYING → UNDEPLOYED (gateway confirms success)
UNDEPLOYING → FAILED (gateway reports failure or timeout)

Acknowledgement Flow

1. Control plane creates deployment → sets status to DEPLOYING and writes performed_at = NOW() to deployment_status
2. Control plane pushes event to gateway via WebSocket (includes deploymentId and performedAt)
3. Gateway processes the deployment
4. Gateway sends deployment.ack message back through the same WebSocket connection with:
   • deploymentId, artifactId, resourceType (api/llmprovider/llmproxy)
   • performedAt (echoed back unchanged)
   • status (success/failed), errorMessage (on failure)
5. Control plane receives ack → compares performedAt against DB; if matching:
   • Failure ack → set FAILED unconditionally with status_reason from errorMessage (overwrites any prior status for this performedAt)
   • Success ack → transition DEPLOYING → DEPLOYED only; discard if status is already FAILED

Scenarios

1. Successful Deployment

Scenario: User deploys an API/LLM Provider/LLM Proxy to a gateway.

Flow: Control plane sets status to DEPLOYING with performed_at = NOW() → pushes event to gateway via WebSocket → gateway processes the deployment → gateway sends deployment.ack with status=success and the same performedAt → control plane compares performedAt, matches, transitions status to DEPLOYED.

sequenceDiagram
    actor User
    participant CP as Control Plane
    participant DB as Database
    participant GW as Gateway

    User->>CP: Deploy API
    CP->>DB: SET status=DEPLOYING, performed_at=NOW()
    CP->>GW: (WebsocketEvent) deploy {deploymentId, performedAt, ...}
    GW->>CP: (REST) GET /api/internal/v1/apis/{apiId}
    CP-->>GW: API definition (zip)
    GW->>GW: Apply config & update policy engine
    GW->>CP: (WebsocketEvent) deployment.ack {deploymentId, performedAt, status=success}
    CP->>DB: SELECT performed_at WHERE artifact+gateway
    DB-->>CP: performed_at (matches)
    CP->>DB: SET status=DEPLOYED
    CP-->>User: Status: DEPLOYED

Loading

2. Successful Undeployment

Scenario: User removes an API/LLM Provider/LLM Proxy from a gateway.

Flow: Control plane sets status to UNDEPLOYING with performed_at = NOW() → pushes undeployment event → gateway removes the resource → gateway sends deployment.ack with status=success → control plane transitions status to UNDEPLOYED.

3. No Acknowledgement (Gateway Timeout)

Scenario: Deployment pushed (status: DEPLOYING) but gateway never responds — could be due to crash, network partition, or processing hang.

Solution: Background job runs every 1 minute, queries for DEPLOYING/UNDEPLOYING entries older than 5 minutes (configurable), and marks them as FAILED.

sequenceDiagram
    participant CP as Control Plane
    participant DB as Database
    participant GW as Gateway
    participant BG as Timeout Job

    CP->>DB: SET status=DEPLOYING, performed_at=NOW()
    CP->>GW: WS event {deploymentId, performedAt, ...}
    note over GW: Gateway crashes / hangs / network lost
    GW--xCP: (no ack)

    loop Every 1 minute
        BG->>DB: SELECT WHERE status IN (DEPLOYING, UNDEPLOYING)<br/>AND performed_at < NOW() - 5min
        DB-->>BG: stale entries
        BG->>DB: SET status=FAILED, status_reason="deployment timed out"
    end

Loading

4. Failed Deployment with Existing Deployed State

Scenario: A second deployment attempt fails while a previous deployment is already running on the gateway.

Solution: deployment_status shows the latest deployment with status=FAILED. The gateway still runs the previous deployment's configuration since the new attempt never succeeded. The user can:

• Retry by creating a new deployment
• Explicitly restore the previous deployment

5. Gateway Disconnect During Deployment

Scenario: Gateway receives the event, starts processing, then disconnects before sending ack.

Solution: The timeout background job catches this and marks as FAILED after the timeout period. On reconnect, the gateway can optionally reconcile its state (future enhancement).

6. New Deployment While Previous is In-Progress

Scenario: User triggers a new deployment while a previous one is still DEPLOYING.

Solution: Allow D2 to proceed. It overwrites the in-progress entry with a new performed_at. When D1's ack arrives carrying the old performed_at, the handler detects a mismatch and discards it. No rejection, no queuing needed.

7. Multiple Gateway Connections (Clustering)

Scenario: A gateway has multiple WebSocket connections (e.g., 3 replicas in a cluster). The deployment event is broadcast to all connections. 1 replica fails, 2 succeed.

Solution: Any failure = FAILED. A failure ack overwrites DEPLOYING or even DEPLOYED to FAILED as long as performed_at matches. A success ack only transitions DEPLOYING → DEPLOYED; if status is already FAILED, the success is discarded. No counters or extra tables needed — the result depends purely on arrival order:

• If the failure arrives first → status = FAILED; the two success acks are discarded
• If a success arrives first → status = DEPLOYED; the failure ack still overwrites to FAILED

Either way, a single failing replica causes the final status to be FAILED.

8. Stale Acknowledgements

Scenario: Gateway sends an ack for a deployment that has already timed out and been marked FAILED, or for a superseded deployment.

Solution: The ack handler compares performedAt from the ack against the current DB value. A mismatch means the ack is stale and is discarded. Logged for observability.

Scope

• REST APIs (api.deployed, api.undeployed)
• LLM Providers (llmprovider.deployed, llmprovider.undeployed)
• LLM Proxies (llmproxy.deployed, llmproxy.undeployed)

Changes Overview

Component	Change
Database	Add DEPLOYING, UNDEPLOYING, FAILED to status CHECK constraint; add performed_at and status_reason columns to deployment_status
Control Plane - WebSocket Handler	Parse incoming gateway messages in readLoop, route deployment.ack to service
Control Plane - Deployment Service	Set DEPLOYING/UNDEPLOYING instead of final status; allow concurrent deployments (new overwrites in-progress)
Control Plane - Deployment Service	Add HandleAck method; handle acknowledgements, update status
Control Plane - Timeout Service	Background job to mark stale in-progress deployments as FAILED
Gateway - WebSocket Client	Add sendMessage/sendDeploymentAck methods with write mutex
Gateway - Event Handlers	Send ack after processing each deployment/undeployment event
Event Payloads	Add deploymentId and performedAt to undeployment events (deployment events already carry deploymentId; add performedAt to all)

Future Work

Deployment History Tracking

Introduce a deployment_events table to record every state transition (DEPLOYING, DEPLOYED, UNDEPLOYING, UNDEPLOYED, FAILED) with timestamps and error messages. This would provide a full audit trail of all deployment attempts per artifact per gateway, allowing users to query the complete history of deployments.

[dushaniw]

+1 for the approach. Minor questions.

1. On Gateway Disconnect During Deployment
   What should the CP do when a gateway disconnects while in DEPLOYING/UNDEPLOYING state? The timeout job would eventually mark it FAILED after 5 minutes. But you already know the gateway is gone (WebSocket connection closed). Cant we transition to FAILED immediately on disconnect rather than waiting for the timeout for a faster feedback for user.

2. What is basis for sending the acks from the GW Controller? Are we depending on the xDS callback for acks or use the UpdateSnapshot return value in the goroutine closure directly ? because accessing the performedAt from the callback wont be possible.

3. Can we use the updated_at current column in deployment_status table for performed_at or is it a new column?

4. From platform-api side, are we allowing for deployment to be undeployed (undeploying) while it is in deploying state or block it until it reaches a deployed or failure?

[malinthaprasan]

+1 for the approach. Minor questions.

1. On Gateway Disconnect During Deployment
   What should the CP do when a gateway disconnects while in DEPLOYING/UNDEPLOYING state? The timeout job would eventually mark it FAILED after 5 minutes. But you already know the gateway is gone (WebSocket connection closed). Cant we transition to FAILED immediately on disconnect rather than waiting for the timeout for a faster feedback for user.

There are some complexities we need to consider when doing this because multiple gateway replicas (controllers) can connect to multiple API Platform replicas in control plane and each communication can happen in different channels. Consider the below scenario.

1. There are two gateways G1, G2 and connected to two platform API replicas P1 P2. Say G1,G2 -> P1 .. P2.
2. API deployment call (REST API) gets routed to P1.
3. P1 sends it to G1 via websocket connection
4. G1 goes down while doing the deployment.
5. Although G1 goes down, internally G1 and G2 will be synced via an HA mechanism. Whatever the gateway runtimes previously connected to G1, can also get updates via G2.
6. G2 deploys the API
7. G2 confirms the deployment to P2. (G2 -> P2 -> DB)
8. G1 starts again later and performs normally

So, in this case even though one gateways stops, still the deployment can happen asynchronously. I guess its a bit hard to isolate only the failure cases and proactively fail immediately.

2. What is basis for sending the acks from the GW Controller? Are we depending on the xDS callback for acks or use the UpdateSnapshot return value in the goroutine closure directly ? because accessing the performedAt from the callback wont be possible.

Had a discussion regarding this (participants: @malinthaprasan @dushaniw @renuka-fernando) and the conclusion was set the deployment before the actual deployment in the gateway runtimes after doing initial validations. As there are multiple gateway replicas etc handing this perfectly is complicated. We'll handle this later if needed.

3. Can we use the updated_at current column in deployment_status table for performed_at or is it a new column?

This is a new column. Since the table row will be updated twice during one deployment () -> Deploying -> Deployed, updated_at will be updated both of the times. But we need an additional column to mark the initial deployment/undeployment time.

4. From platform-api side, are we allowing for deployment to be undeployed (undeploying) while it is in deploying state or block it until it reaches a deployed or failure?

I think we should let them retry otherwise during a temporary lost connection state, a user may not be able to retry deployment during the timeout period (5 min). But we should clearly show the current deploying state in the UI while allowing the redeployment.

[malinthaprasan]

Update: Separating Desired State from Actual State

Had a further discussion regarding how we handle the deployment state in the platform API (participants: @malinthaprasan @renuka-fernando @dushaniw)

The status column seems to be doing double duty - representing both what the user intended (desired state) and what is actually happening (current state). For example, when a user deploys an API, the desired state is DEPLOYED, but the actual state progresses through DEPLOYING → DEPLOYED | FAILED.

To make this explicit, we're introducing a status_desired column:

• status_desired — set once on user action; values: DEPLOYED, UNDEPLOYED
• status — evolves through the deployment lifecycle: DEPLOYING → DEPLOYED | FAILED, UNDEPLOYING → UNDEPLOYED | FAILED

This separation makes it easier to answer two distinct questions independently:

• "What did the user want?" : status_desired
• "What actually happened?" : status

When a gateway restarts and requests to fetch the deployments it needs, it can rely on the status_desired column without being worried about the status column.

[malinthaprasan]

Release Steps

The production deployment_status table is live and the UI reads status directly. Introducing new status values must be done incrementally to avoid breaking existing behaviour.

Step 1 — Database Migration

Additive schema changes only, no code deployed.

• Add three new nullable columns to deployment_status:

  ALTER TABLE deployment_status ADD COLUMN status_desired TEXT;
  ALTER TABLE deployment_status ADD COLUMN performed_at DATETIME;
  ALTER TABLE deployment_status ADD COLUMN status_reason TEXT;

• Backfill existing rows:

  UPDATE deployment_status SET status_desired = status;
  UPDATE deployment_status SET performed_at = updated_at;

Step 2 — UI Backward-Compatibility Update

Must be released before Step 5.

• Treat DEPLOYING as DEPLOYED for display purposes
• Treat UNDEPLOYING as UNDEPLOYED for display purposes

Step 3 — Code Changes

Platform-API:

• Write DEPLOYING/UNDEPLOYING on deployment initiation instead of the final status
• Handle incoming deployment.ack messages from the gateway
• Expose status_desired, status_reason, and performed_at in API responses
• Implement timeout background job with two config flags: enable/disable, and fallback mode (optimistic → DEPLOYED/UNDEPLOYED, strict → FAILED)

Gateway:

• Send deployment.ack message back to the control plane after processing each deployment/undeployment event

Step 4 — Release Gateway

• Release the new gateway version that supports sending deployment.ack messages back to the control plane

Step 5 — Release and Deploy Platform-API

Prerequisite: Step 2 UI must already be in production.

• Release and deploy the new platform-api version
• Enable the timeout job in optimistic mode — timed-out deployments resolve to DEPLOYED/UNDEPLOYED rather than FAILED

Step 6 — UI Final Update

• Show DEPLOYING/UNDEPLOYING as distinct in-progress states
• Display FAILED with status_reason for error messaging
• Poll deployment status until a terminal state is reached
• Use status_desired to show user intent alongside actual status
• Update the gateway version referenced in the UI to the version released in Step 4
• Remove the Step 2 compatibility shim

Step 7 — Switch Timeout Job to Strict Mode

• Update the timeout job configuration to strict
• Timed-out deployments are now marked FAILED with status_reason = "deployment timed out"