Proposal: Gateway Deployment Sync on Startup

[dushaniw]

Problem

Gateways restart with stale configurations because WebSocket events are missed while the gateway is down. The platform-api has newer deployments that the gateway doesn't know about, leading to:

• Orphaned configs (undeployed APIs still being served)
• Configuration drift over time

Proposed Solution

Async sync on gateway startup — Gateway fetches expected deployment state from platform-api and reconciles locally.

Key Design Decisions

Decision	Choice	Rationale
Who computes the diff?	Gateway (not platform-api)	Gateways scale out more than platform-api. Computing diff server-side would create a bottleneck with N gateways.
Sync mode	Async (background)	Serving stale data temporarily is acceptable. Gateway independence is prioritized—gateway starts immediately and syncs in background.
What to track?	deployment_id column	Exact artifact version matching. Simpler than content hashing and deployment ID is already available.
Sync triggers	Startup only (Phase 1)	WebSocket reconnect sync and periodic sync deferred—can revisit based on operational experience.
Race conditions	Idempotent operations	If WebSocket event arrives during sync, newer deployment naturally wins since deploymentId is unique.
Batch vs individual fetch	Batch fetch	Single request for all missing deployments avoids N round-trips.

Required Changes

Gateway Controller - DB Schema

ALTER TABLE deployments ADD COLUMN deployment_id TEXT;
CREATE INDEX IF NOT EXISTS idx_deployment_id ON deployments(deployment_id);

Platform API - New Internal Endpoints

1. List Expected Deployments

GET /api/internal/v1/gateways/{gatewayId}/deployments

Response:
{
  "deployments": [
    { "apiId": "abc-123", "deploymentId": "dep-789", "artifactType": "rest-api" },
    { "apiId": "xyz-456", "deploymentId": "dep-456", "artifactType": "rest-api" },
    { "apiId": "llm-001", "deploymentId": "dep-111", "artifactType": "llm-provider" }
  ]
}

2. Batch Fetch Deployment Content

POST /api/internal/v1/gateways/{gatewayId}/deployments/batch

Request:
{
  "deploymentIds": ["dep-789", "dep-456", "dep-111"]
}

Response: application/zip

Zip Structure:

batch.zip
├── dep-789/
│   └── api.yaml
├── dep-456/
│   └── api.yaml
└── dep-111/
    └── api.yaml

Each top-level directory is named by deploymentId, allowing the gateway to identify which content belongs to which deployment.

GW High-Level Flow

Startup
        ↓
Load from local DB
         ↓
Start serving traffic
          ↓
Background sync
          ↓
GET /deployments (list expected)
          ↓
Compute diff locally
          ↓
POST /deployments/batch (fetch missing)
          ↓
Extract zip, deploy each, remove orphaned

Phase 1 Scope

REST APIs, LLM Providers, LLM Proxies
Startup sync (async mode)
Batch fetch for efficiency

[dushaniw]

Updated Design: Refinements based on team discussion

After further analysis and team discussions with @malinthaprasan and @VirajSalaka , here are the key refinements to the initial proposal:

---

1. Enhanced API Response

The /deployments endpoint response now includes additional fields:

{
  "deployments": [
    { 
      "apiId": "abc-123", 
      "deploymentId": "dep-789", 
      "kind": "RestAPI",
      "state": "DEPLOYED",
      "deployedAt": "2026-03-04T10:30:00Z"
    }
  ]
}

Field	Purpose
kind	Artifact type: RestAPI, LLMProvider, LLMProxy
state	DEPLOYED or UNDEPLOYED - determines gateway action
deployedAt	Timestamp for race condition resolution

---

2. Sync Actions Clarified

Based on comparing platform state vs local state:

Scenario	Gateway Action
New deployment (not in local)	Fetch content, insert
Deployment ID changed	Fetch new content, update
Only status changed	Update status_desired (no content fetch)
Not in platform list	DELETE from gateway (API was deleted from platform)

Note: UNDEPLOYED status updates status only (keeps in DB, removes from xDS). Only APIs deleted from platform are deleted from gateway DB.

---

3. Race Condition Handling: Timestamp-Guarded Writes

To handle races between startup sync and WebSocket events:

INSERT INTO artifacts (api_id, deployment_id, ..., deployed_at)
VALUES (?, ?, ..., ?)
ON CONFLICT (api_id) DO UPDATE SET ...
WHERE artifacts.deployed_at < EXCLUDED.deployed_at

Key rule: Only publish to EventHub if rowsAffected > 0 - prevents stale events from propagating.

---

4. Integration with HA Sync (EventHub)

Startup sync uses the same abstraction being developed for Gateway Controller HA:

Sync/WebSocket Event
        │
        ▼
Timestamp-guarded DB write
        │
        ├── rowsAffected > 0 → Publish to EventHub → ConfigStore → xDS
        │
        └── rowsAffected = 0 → Skip (stale data)

Self-published events: Controllers update their own ConfigStore via EventHub (clean single path for all updates).

---

5. Chunked Batch Fetching

To handle large number of deployments:

• Gateway-side chunking (configurable batch_size, default: 50)
• Platform-API stays simple - no pagination needed
• If a chunk fails, log and continue with next chunk
• Partial failures: log and continue, retry on next restart

Configuration:

[controller.sync]
batch_size = 50

---

6. DB Schema

Builds on Discussion #1311:

ALTER TABLE artifacts ADD COLUMN deployment_id TEXT;
CREATE INDEX IF NOT EXISTS idx_deployment_id ON artifacts(deployment_id);

Column	Source
deployment_id	New (this PR)
deployed_at	From #1311 (used for timestamp guard)
kind	From #1311
status_desired	From #1311 (renamed from status)

---

7. EventHub Ordering Assumption

ConfigStore does NOT have timestamp guard - relies on ordered EventHub delivery.

This is safe as long as:

• Single scheduler instance
• Events table ordered by insertion time
• FIFO processing
• No retry/redelivery

Discussion point: If EventHub architecture changes (multiple schedulers, parallel processing), ConfigStore should add timestamp guard.

---

Phase 1 Scope (Updated)

• Add deployment_id column to artifacts table
• Implement GET /internal/v1/gateways/{id}/deployments (list with kind, state, deployedAt)
• Implement POST /internal/v1/gateways/{id}/deployments/batch (batch fetch)
• Gateway sync logic with chunked fetching
• Timestamp-guarded writes for all operations
• Integration with HA sync abstraction (EventHub)
• Configurable batch_size
• Support REST APIs, LLM Providers, LLM Proxies

Dependencies:

• Gateway Controller HA sync (EventHub) - in progress
• DB restructuring from [Proposal] Gateway Controller DB Schema — Per-Resource-Type Tables #1311

---

Open Discussion Points

1. EventHub scaling: If we add multiple schedulers in future, ConfigStore needs timestamp guard
2. Status update abstraction: May need to add timestamp-guarded status update method to HA sync implementation

---

Full design details: docs/proposals/gateway-deployment-sync-discussion.md