Skip to content

feat(bootstrap): add runner health guards #1312

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
casey-brooks wants to merge 8 commits into
base: main
Choose a base branch
from
+1,289 −241
Open

feat(bootstrap): add runner health guards #1312

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
c815b01
feat(bootstrap): add runner health guards
casey-brooks Feb 21, 2026
27362c1
feat(server): finalize optional docker runner bootstrap
casey-brooks Feb 21, 2026
ac443b1
fix(server): type docker connectivity contract
casey-brooks Feb 21, 2026
8e81758
fix(server): emit json terminal 503 response
casey-brooks Feb 21, 2026
d69c57b
test(server): add production bootstrap di regression
casey-brooks Feb 21, 2026
fc39775
fix(bootstrap): ensure ConfigService injection for DockerRunnerStatus…
casey-brooks Feb 21, 2026
eeec952
test(bootstrap): skip LLM provisioner in production DI test; guard in…
casey-brooks Feb 21, 2026
68f6399
test(bootstrap): skip DB initialization in production DI test
casey-brooks Feb 21, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -188,6 +188,19 @@ Key environment variables (server) from packages/platform-server/.env.example an
- DOCKER_RUNNER_BASE_URL (required; default http://docker-runner:7071)
- DOCKER_RUNNER_SHARED_SECRET (required HMAC credential)
- DOCKER_RUNNER_TIMEOUT_MS (optional request timeout; default 30000)
- DOCKER_RUNNER_OPTIONAL (default true; set to false to keep fail-fast bootstrap)
- DOCKER_RUNNER_CONNECT_RETRY_BASE_DELAY_MS (default 500)
- DOCKER_RUNNER_CONNECT_RETRY_MAX_DELAY_MS (default 30000)
- DOCKER_RUNNER_CONNECT_RETRY_JITTER_MS (default 250)
- DOCKER_RUNNER_CONNECT_PROBE_INTERVAL_MS (default 30000 while runner is healthy)
- DOCKER_RUNNER_CONNECT_MAX_RETRIES (default 0 → unlimited background retries)
- Volume GC:
- VOLUME_GC_ENABLED (default true)
- VOLUME_GC_INTERVAL_MS (default 60000)
- VOLUME_GC_MAX_PER_SWEEP (default 100)
- VOLUME_GC_CONCURRENCY (default 3)
- VOLUME_GC_COOLDOWN_MS (default 600000)
- VOLUME_GC_SWEEP_TIMEOUT_MS (default 15000)
- Nix/NCPS:
- NCPS_ENABLED (default false)
- NCPS_URL_SERVER, NCPS_URL_CONTAINER (default http://ncps:8501)
Expand Down
2 changes: 1 addition & 1 deletion docs/product-spec.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -116,7 +116,7 @@ Configuration matrix (server env vars)
- VAULT_ENABLED: true|false (default false)
- VAULT_ADDR, VAULT_TOKEN
- DOCKER_MIRROR_URL (default http://registry-mirror:5000)
- DOCKER_RUNNER_BASE_URL, DOCKER_RUNNER_SHARED_SECRET (required for docker-runner), plus optional DOCKER_RUNNER_TIMEOUT_MS (default 30000).
- DOCKER_RUNNER_BASE_URL, DOCKER_RUNNER_SHARED_SECRET (required for docker-runner), plus optional DOCKER_RUNNER_TIMEOUT_MS (default 30000), DOCKER_RUNNER_OPTIONAL (default true; set false to fail-fast), and DOCKER_RUNNER_CONNECT_* knobs (RETRY_BASE_DELAY_MS=500, RETRY_MAX_DELAY_MS=30000, RETRY_JITTER_MS=250, PROBE_INTERVAL_MS=30000, MAX_RETRIES=0 for unlimited background retries).
- MCP_TOOLS_STALE_TIMEOUT_MS
- LANGGRAPH_CHECKPOINTER: postgres (default)
- POSTGRES_URL (postgres connection string)
Expand Down
4 changes: 4 additions & 0 deletions docs/technical-overview.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -71,11 +71,15 @@ Remote Docker runner
- The runner exposes authenticated Fastify HTTP/SSE/WebSocket endpoints with HMAC headers derived solely from `DOCKER_RUNNER_SHARED_SECRET`.
- Only the docker-runner service mounts `/var/run/docker.sock` in default stacks; platform-server and auxiliary services talk to it over the internal network (default http://docker-runner:7071).
- Container events are forwarded via SSE so the existing watcher pipeline (ContainerEventProcessor, cleanup jobs, metrics) remains unchanged.
- Connectivity is tracked by a background `DockerRunnerConnectivityMonitor` that polls `/v1/ready` with exponential backoff (base-delay, max-delay, jitter, probe interval, and optional retry cap are configurable via DOCKER_RUNNER_CONNECT_* env vars).
- When `DOCKER_RUNNER_OPTIONAL=true` (default) the server continues booting even if the runner is unreachable; when set to `false` the first failed probe aborts bootstrap (legacy fail-fast mode).
- The monitor streams status into `DockerRunnerStatusService`, which feeds `/health`, Volume GC, REST guards, and terminal/websocket gating. Terminals and container APIs short-circuit with `docker_runner_not_ready` until status returns `up`.

Defaults and toggles
- LiveGraphRuntime serializes apply operations by default.
- PRTrigger intervalMs default 60000; includeAuthored default false.
- MCP restart defaults: maxAttempts 5; backoffMs 2000.
- Docker runner monitor defaults: optional=true, retry base delay 500ms, max delay 30s, jitter 250ms, probe interval 30s when healthy, max retries 0 (infinite).

How to Develop & Test
- Prereqs: Node.js 20+, pnpm 9+, Docker, Postgres
Expand Down
1 change: 1 addition & 0 deletions packages/platform-server/.env.example
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,7 @@ DOCKER_RUNNER_SHARED_SECRET=dev-shared-secret
# VOLUME_GC_MAX_PER_SWEEP=100
# VOLUME_GC_CONCURRENCY=3
# VOLUME_GC_COOLDOWN_MS=600000
# VOLUME_GC_SWEEP_TIMEOUT_MS=15000

# Container retention window (in days). Set to 0 to retain indefinitely.
CONTAINERS_RETENTION_DAYS=14
1 change: 1 addition & 0 deletions packages/platform-server/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@ Graph persistence
- `VOLUME_GC_MAX_PER_SWEEP` (default `100`)
- `VOLUME_GC_CONCURRENCY` (default `3`)
- `VOLUME_GC_COOLDOWN_MS` (default `600000`)
- `VOLUME_GC_SWEEP_TIMEOUT_MS` (default `15000`)
-
## MCP environment configuration

Expand Down
Loading
Loading