Service health indicators in WebUI

[junhaoliao]

Request

Display health indicators for CLP package services in the WebUI to help users identify when critical backend components are unavailable.

Problem

Users have experienced situations where the WebUI is accessible and appears functional, but queries are not being processed because the query job orchestration components (query-scheduler, query-worker, reducer, etc.) are down. While administrators can check service health via orchestrator tools (docker compose ps, kubectl get pods), there is no visibility into service health from user-facing interfaces like the WebUI, making it difficult for users to diagnose such issues.

Why this matters

• User Experience: Users cannot easily determine why their queries are not processing
• Debugging: Without health indicators, users have no visibility into which components might be failing
• Operational Awareness: Administrators need to quickly identify service outages without manually checking container/pod status

Affected services

Based on the current architecture in tools/deployment/package/docker-compose-all.yaml and tools/deployment/package-helm/templates/, the services that need health monitoring include:

Note: Current health checks are defined per orchestrator:

• Docker Compose: healthcheck blocks in docker-compose-all.yaml
• Kubernetes (Helm): readinessProbe / livenessProbe in deployment templates

Core job orchestration services

Service	Description	Port	Current Health Endpoint
query-scheduler	Schedules query jobs	7000	None (the port is only a TCP listener for reducers)
compression-scheduler	Schedules compression jobs	-	None
query-worker	Celery worker for executing queries	-	None (Celery process)
compression-worker	Celery worker for executing compression jobs	-	None (Celery process)
reducer	Aggregates query results	-	None

Supporting services

Service	Description	Port	Current Health Endpoint
api-server	REST API server	3001	GET /health
webui	Web interface	4000	TCP socket check
garbage-collector	Cleans up old archives and results	-	None
mcp-server	MCP server (optional)	8000	GET /health
log-ingestor	Ingestion service	3002	GET /health

Third-party services

Service	Description	Port	Health Check Method
database	MariaDB	3306	mysqladmin ping
queue	RabbitMQ	5672	rabbitmq-diagnostics check_running
redis	Redis	6379	redis-cli PING
results-cache	MongoDB	27017	mongosh ping

Initialization jobs

Service	Description	Health Check Method
db-table-creator	Creates database tables in MariaDB	Job completion status (one-time)
results-cache-indices-creator	Initializes MongoDB indices	Job completion status (one-time)

Note: Most services depend on these initialization jobs completing successfully before starting.

Possible implementation

Two decisions need to be made:

1. How services report health — the mechanism for collecting health status from services
2. How health statuses are cached/exposed — how the API server stores and exposes aggregated health data

1. Alternative approaches for health reporting

How should services report their health status to a central aggregator?

Option	Description	Advantages	Disadvantages
1A: Orchestrator-based	Leverage Docker/Kubernetes APIs to get container/pod health status	• Uses existing health checks defined in compose/helm files	• Not orchestrator agnostic • Docker: requires exposing socket (security concern) • Kubernetes: requires additional RBAC permissions
1B: Services send heartbeats to API server (recommended)	Services periodically POST health reports to API server	• Orchestrator agnostic • Services only need to make HTTP requests (simpler than serving) • Single aggregation point as source of truth	• Requires adding HTTP client to each service
1C: API server scrapes services	API server periodically polls each service's health endpoint	• Similar to Prometheus model • Bypasses orchestrator	• Requires all services to expose HTTP endpoints (not all are HTTP servers) • Requires service discovery (API server needs to know hostnames assigned by Docker Compose / Kubernetes)

Option 1B implementation details:

1. Add a POST /health endpoint to the API server that accepts health reports
2. Each service periodically (e.g., every 10 seconds) sends a report with:
   • Service name
   • Service instance ID
   • Optional timestamp (for debugging clock skew / network delays; API server's receive time is authoritative for health calculations)
   • Optional error message to explicitly mark as unhealthy (e.g., "failed to connect to database")
   • Optional status details (e.g., queue depth, active jobs)
3. API server also marks services as unhealthy if no heartbeat received within a threshold (e.g., 30 seconds)

2. Alternatives for health status storage/caching

Some entity (now we assume the API server) aggregates health statuses. Options for how it
stores/exposes them:

Option	Description	Advantages	Disadvantages
2A: API server in-memory cache (recommended)	Cache in memory, expose via GET /health	• Simplest; no external storage • Health data is ephemeral by nature • Orchestrators can query for health checks	• WebUI must poll (no push updates) • Data lost on restart (acceptable)
2B: MongoDB (results-cache)	Store in dedicated MongoDB collection	• WebUI can use CDC via Socket.IO for real-time updates (existing pattern)	• Additional complexity for ephemeral data
2C: Redis	Store with TTL-based expiry	• Fast reads/writes • TTL auto-expires stale entries	• WebUI doesn't connect to Redis • Requires new infrastructure
2D: MariaDB / MySQL (clp-db)	Store in heartbeat table	• WebUI already connects to clp-db • Transactional consistency	• WebUI must poll (no CDC) • Additional load on primary database

Option 2A endpoints:

• GET /health — returns health status of all services (for WebUI)
• GET /health?service=<name>&instance=<id> — returns health status of a specific service instance (for container orchestrator health checks on services without their own endpoints)

Caveat for orchestrator health checks: This creates a chicken-and-egg problem. Currently, API server has hard dependencies (depends_on database / initContainers waiting for db-table-creator), so it can't start before other services without relaxing these dependencies.

Recommended architecture (Option 1B + 2A)

flowchart LR
    subgraph Services
        QS[query-scheduler]
        CS[compression-scheduler]
        QW[query-worker]
        CW[compression-worker]
        R[reducer]
        GC[garbage-collector]
    end

    subgraph Aggregator
        API[API Server<br/>in-memory cache]
    end

    subgraph Frontend
        WebUI[WebUI]
    end

    subgraph Orchestrator
        DC[Docker Compose /<br/>Kubernetes]
    end

    QS -->|POST /health| API
    CS -->|POST /health| API
    QW -->|POST /health| API
    CW -->|POST /health| API
    R -->|POST /health| API
    GC -->|POST /health| API

    API -->|GET /health| WebUI
    API -->|GET /health?service=X| DC

Loading

Implementation steps (Option 1B + 2A)

1. API server changes:

   • Add POST /health endpoint to accept service health reports
   • Add GET /health endpoint to return aggregated health status of all services (for WebUI)
   • Add GET /health?service=<name>&instance=<id> for querying specific service health (for container orchestrator health checks)
   • Add background task to mark services as unhealthy if no report received within threshold
   • Cache health statuses in memory

2. Service changes (first-party):

   • Add health report HTTP client to each long-running Python service (query-scheduler, compression-scheduler, reducer, garbage-collector)
   • Add health report mechanism to Celery workers (query-worker, compression-worker)
   • Configure report interval via environment variable or config
   • For initialization jobs (db-table-creator, results-cache-indices-creator): report completion status once upon success/failure

3. Third-party service health reporting (optional, future enhancement):

   Third-party services (database, queue, redis, results-cache) don't run our code, so they can't directly report health. Possible approaches:

   • Extend existing healthchecks: Append a curl command to existing healthcheck scripts, e.g.,
     mysqladmin ping && curl -X POST http://api-server:3001/health -d '{"service":"database"}'

4. WebUI changes:

   • Poll GET /health endpoint periodically (e.g., every 5 seconds)
   • Add health status display component (e.g., status bar or dedicated page)
   • Visual indicators: green (healthy), red (unhealthy/missing); optionally yellow (degraded) in future

5. Container orchestrator changes (optional, requires more thought):

   • For services without their own health endpoints, configure health checks to use GET /health?service=<name>&instance=<id> on the API server (see caveat in Option 2A above)

Configuration options

# Example values.yaml additions
clpConfig:
  # Per-service config to optionally enable reporting (so services don't have a hard dependency on API server)
  query_scheduler:
    health_reporting:
      enabled: true           # optional; service continues to function if API server is unavailable
      interval: 10            # seconds between reports
      unhealthy_threshold: 30 # seconds without report before API server marks as unhealthy

  compression_scheduler:
    health_reporting:
      enabled: true
      interval: 10
      unhealthy_threshold: 30

  # ... similar for other services

References

• Docker Compose health checks: tools/deployment/package/docker-compose-all.yaml
• Helm deployment templates: tools/deployment/package-helm/templates/
• API server routes: components/api-server/src/routes.rs
• WebUI MongoDB integration: components/webui/server/src/plugins/app/socket/MongoSocketIoServer/
• Query scheduler: components/job-orchestration/job_orchestration/scheduler/query/query_scheduler.py

[coderabbitai]

📝 CodeRabbit Plan Mode

Generate an implementation plan and prompts that you can use with your favorite coding agent.

• Create Plan

Examples

• Example 1
• Example 2

_{You can configure auto-planning by selecting labels in the issue_enrichment configuration.}

🔗 Similar Issues

Related Issues

• Add health check endpoints/mechanisms for compression scheduler and worker #1390
• Migrate start_clp.py to Docker Compose #1177
• Add healthcheck to api-server service in Docker Compose #1608

👤 Suggested Assignees

• coderabbitai
• junhaoliao

---

🧪 Issue enrichment is currently in open beta.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord or schedule a call!

[coderabbitai]

Implementation Plan

Approach

Implement a heartbeat-based health monitoring system where Python services periodically report their health status to the API server, which caches this information in memory and exposes it via REST endpoints for the WebUI to display to users.

Observations

The CLP system consists of a Rust-based API server (using Axum framework), Python-based job orchestration services (using asyncio and Celery), and a React-based WebUI (using Ant Design and Fastify server). Services are deployed via Docker Compose or Kubernetes Helm charts. Configuration is managed through YAML files with Pydantic validation models. The API server currently has a simple GET /health endpoint; Python services use asyncio for concurrency; and the WebUI uses React Query for API state management with Socket.IO for real-time MongoDB updates.

Assumptions

Assumption 1: Service Instance Identification

Options Considered:

• Use HOSTNAME environment variable (set by Docker/Kubernetes to container ID or pod name)
• Generate UUID on service startup
• Use hostname + process ID combination

Chosen Option: Use the HOSTNAME environment variable, as Docker and Kubernetes automatically set this to the container ID or pod name respectively

Rationale: This provides unique identification without additional configuration and aligns with standard orchestrator practices.

Assumption 2: HTTP Client Library for Python Services

Options Considered:

• Use requests for all services (widely adopted, synchronous only)
• Use aiohttp for all services (requires async context)
• Mix: aiohttp for async services, requests for synchronous services and Celery workers

Chosen Option: Use aiohttp for async services (query-scheduler, garbage-collector, reducer) and requests for synchronous services (compression-scheduler) and Celery workers

Rationale: This aligns with each service's concurrency model and avoids forcing async where it doesn't naturally fit.

Assumption 3: Background Heartbeat Task Implementation

Options Considered:

• Integrate into existing async event loops for async services
• Use separate threads for all services
• Use multiprocessing for isolation

Chosen Option: For async services, create an asyncio task within the existing event loop. For synchronous services and Celery workers, use threading.Thread with daemon=True

Rationale: This follows the natural concurrency patterns already established in each service type.

💡 User Tips

Regenerate the plan with different choices with @coderabbitai <feedback>.

Plan

Phase 1: Core Health Infrastructure

This phase establishes the foundation for health monitoring by extending the configuration schema and implementing the API server's health management capabilities. The API server will maintain an in-memory cache of service health states and expose endpoints for both receiving heartbeats and querying health status.

Task 1: Extend Configuration Schema

Add health reporting configuration to the Pydantic models in clp-py-utils.

• Create a new HealthReporting Pydantic model in components/clp-py-utils/clp_py_utils/clp_config.py with fields: enabled: bool = True, api_server_url: str, interval_seconds: PositiveFloat = 10.0, timeout_seconds: PositiveFloat = 5.0
• Add health_reporting: HealthReporting | None = None field to existing QueryScheduler, CompressionScheduler models
• Create new models for Reducer and GarbageCollector (if they don't exist) with health_reporting field
• Update ClpConfig root model to include these new service configurations if needed
• Follow existing pattern where optional configurations use Field | None = None with default instances

Task 2: Implement API Server Health Cache

Add in-memory health status storage to the API server using Rust's concurrency primitives.

• Create a new module in components/api-server/src/ (e.g., health_manager.rs) containing a HealthManager struct
• Use Arc<DashMap<String, HealthStatus>> or Arc<RwLock<HashMap<String, HealthStatus>>> for thread-safe cache (key format: "{service_name}:{instance_id}")
• Define HealthStatus struct with fields: service_name: String, instance_id: String, last_heartbeat: Instant, status: HealthState (enum: Healthy/Unhealthy), error_message: Option<String>, status_details: Option<Value>
• Implement methods: update_heartbeat(), get_service_health(), get_all_health(), mark_stale_as_unhealthy(threshold: Duration)
• Add HealthManager instance to the shared Client state struct in components/api-server/src/client.rs

Task 3: Add Health Endpoints to API Server

Implement REST endpoints for health management following Axum's routing patterns.

• Add POST /health endpoint in components/api-server/src/routes.rs accepting JSON body with service_name, instance_id, timestamp (required), and optional error_message, status_details fields
• Add GET /health endpoint returning JSON array of all service health statuses
• Add GET /health?service={name}&instance={id} variant returning single service health status
• Use Utoipa macros (#[utoipa::path]) for OpenAPI documentation
• Extract State<Client> to access the HealthManager
• Return appropriate HTTP status codes (200 for healthy, 503 for unhealthy services)

Task 4: Add Background Cleanup Task

Create a background task to mark services as unhealthy when heartbeats stop.

• Add a Tokio task in components/api-server/src/bin/api_server.rs that runs every 10 seconds
• Call health_manager.mark_stale_as_unhealthy(Duration::from_secs(30)) to mark services unhealthy if no heartbeat received within threshold
• Use tokio::time::interval for periodic execution
• Ensure task is spawned after server initialization and cancelled during graceful shutdown

🤖 Prompt for AI agents

Establish the core health monitoring infrastructure in the API server. This
phase creates the foundation for tracking service health states.

Extend the Pydantic configuration schema in clp-py-utils:
- Create HealthReporting model with enabled, api_server_url, interval_seconds,
and timeout_seconds fields
- Add health_reporting field to QueryScheduler and CompressionScheduler models
- Create Reducer and GarbageCollector models with health_reporting field

Implement API server health cache:
- Create health_manager.rs module with HealthManager struct
- Use thread-safe cache (Arc<DashMap> or Arc<RwLock<HashMap>>)
- Define HealthStatus struct with service_name, instance_id, last_heartbeat,
status, error_message, and status_details
- Implement update_heartbeat(), get_service_health(), get_all_health(), and
mark_stale_as_unhealthy() methods
- Add HealthManager to Client state struct

Add health endpoints:
- POST /health endpoint accepting service_name, instance_id, timestamp,
error_message, and status_details
- GET /health endpoint returning all service health statuses
- GET /health with query parameters for specific service/instance
- Use Utoipa macros for OpenAPI documentation
- Return appropriate status codes (200 for healthy, 503 for unhealthy)

Add background cleanup task:
- Create Tokio task running every 10 seconds to mark stale services as unhealthy
- Use 30-second threshold for stale detection
- Handle graceful shutdown

Phase 2: Service Health Reporting

This phase adds heartbeat reporting capabilities to all Python services. Each service will periodically send health reports to the API server, with implementation details varying based on whether the service uses asyncio or synchronous execution.

Task 1: Create Shared Health Reporter Utility

Build a reusable health reporting component in the shared Python utilities.

• Create new module components/clp-py-utils/clp_py_utils/health_reporter.py
• Implement AsyncHealthReporter class using aiohttp with methods: start(), stop(), report_healthy(), report_unhealthy(error_message)
• Implement SyncHealthReporter class using requests and threading.Thread (daemon=True) for periodic reporting
• Both classes should accept configuration (API URL, service name, interval, timeout) in constructor
• Handle connection failures gracefully (log warnings but don't crash service)
• Use HOSTNAME environment variable for instance_id (with fallback to socket.gethostname())

Task 2: Add Heartbeat to Query Scheduler

Integrate health reporting into the async query scheduler service.

• Import AsyncHealthReporter in components/job-orchestration/job_orchestration/scheduler/query/query_scheduler.py
• Initialize reporter in the main() function after configuration loading, passing health_reporting config from ClpConfig.query_scheduler
• Create asyncio task for reporter using asyncio.create_task(reporter.start()) alongside existing job handler and reducer handler tasks
• Include reporter task in the asyncio.wait() call with FIRST_COMPLETED strategy
• Stop reporter during shutdown alongside other tasks
• Make reporting optional: skip if health_reporting.enabled is False or config is None

Task 3: Add Heartbeat to Compression Scheduler

Integrate health reporting into the synchronous compression scheduler service.

• Import SyncHealthReporter in components/job-orchestration/job_orchestration/scheduler/compression/compression_scheduler.py
• Initialize reporter after configuration loading in the initialization section (before main loop)
• Call reporter.start() to begin background thread before entering main polling loop
• Call reporter.stop() in the SIGTERM handler or cleanup section
• Check health_reporting.enabled flag to conditionally enable

Task 4: Add Heartbeat to Garbage Collector

Integrate health reporting into the async garbage collector service.

• Import AsyncHealthReporter in components/job-orchestration/job_orchestration/garbage_collector/garbage_collector.py
• Initialize reporter in main() after config loading
• Add reporter task to the list of garbage collection tasks monitored by asyncio.wait()
• Ensure reporter is stopped when any collection task fails or during shutdown
• Make optional based on configuration

Task 5: Add Heartbeat to Reducer Service

Integrate health reporting into the reducer's async architecture.

• Import AsyncHealthReporter in components/job-orchestration/job_orchestration/scheduler/query/query_scheduler.py (reducer handler is managed by query scheduler)
• Initialize a separate reporter instance for the reducer with service name "reducer"
• Create reporter task alongside the existing reducer handler task
• Include in the task monitoring and shutdown logic
• Configure via query_scheduler's health_reporting config (or add separate reducer config)

Task 6: Add Heartbeat to Celery Workers

Implement health reporting using Celery's worker lifecycle signals.

• Create new file components/job-orchestration/job_orchestration/executor/health_reporter.py with worker-specific reporter
• Import SyncHealthReporter and Celery signals in the new module
• Use @signals.worker_ready.connect to start reporter thread when worker starts
• Use @signals.worker_shutdown.connect to stop reporter thread when worker stops
• Import and initialize in both components/job-orchestration/job_orchestration/executor/query/celery.py and compress/celery.py
• Load health_reporting config from the Celery app's configuration
• Use service names "query-worker" and "compression-worker" respectively

🤖 Prompt for AI agents

Add heartbeat reporting to all Python services. This phase implements periodic
health status reporting from services to the API server.

Create shared health reporter utility:
- Implement AsyncHealthReporter class using aiohttp with start(), stop(),
report_healthy(), and report_unhealthy() methods
- Implement SyncHealthReporter class using requests and threading.Thread
(daemon=True)
- Accept API URL, service name, interval, and timeout in constructor
- Use HOSTNAME environment variable for instance_id with fallback to
socket.gethostname()
- Handle connection failures gracefully

Integrate health reporting into services:
- Query Scheduler: Initialize AsyncHealthReporter in main(), create asyncio
task, include in asyncio.wait() with FIRST_COMPLETED, handle shutdown
- Compression Scheduler: Initialize SyncHealthReporter, call start() before main
loop, call stop() in cleanup
- Garbage Collector: Initialize AsyncHealthReporter in main(), add to task list
monitored by asyncio.wait()
- Reducer Service: Initialize separate AsyncHealthReporter instance with service
name "reducer", create task alongside reducer handler
- Celery Workers: Create health_reporter.py module, use
@signals.worker_ready.connect and @signals.worker_shutdown.connect, initialize
in query and compress celery.py files

Make all health reporting optional based on configuration

Phase 3: WebUI Health Display

This phase adds health status visualization to the WebUI by implementing API polling in the server and creating UI components in the client to display service health indicators to users.

Task 1: Add Health API Client to WebUI Server

Implement API server communication for health status in the Fastify-based WebUI server.

• Create new route file components/webui/server/src/routes/api/health/index.ts
• Define Fastify route GET /api/health that proxies to API server's GET /health endpoint using the HTTP client
• Parse and transform response if needed to match WebUI's expected format
• Register route in the server's route autoload system (should be picked up automatically from routes directory)
• Handle connection errors gracefully and return appropriate HTTP status codes

Task 2: Create Health Status Hook

Add React Query-based polling for health status in the WebUI client.

• Create new hook file components/webui/client/src/api/useHealthStatus.ts
• Use useQuery from React Query with refetchInterval: 5000 for automatic 5-second polling
• Query the /api/health endpoint via the WebUI server
• Define TypeScript interfaces for health status response matching API server's format
• Return loading state, error state, and health data for components to consume

Task 3: Create Health Status Display Component

Build a visual health indicator component using Ant Design components.

• Create new component file components/webui/client/src/components/HealthStatus/index.tsx
• Use Ant Design's Tag, Badge, or Alert components for status indicators
• Map service health states to colors: green (healthy), red (unhealthy), gray (unknown/stale)
• Display service name, status, and optional error message
• Support both compact (badge) and detailed (list) display modes
• Make component responsive and accessible

Task 4: Integrate Health Display into UI

Add health status visibility to the main application interface.

• Update components/webui/client/src/App.tsx or main layout component to include health status
• Add health status badge/indicator to application header or navigation area
• Consider adding a dedicated health status page/modal for detailed view
• Use the useHealthStatus hook to fetch data
• Show loading state during initial fetch
• Display critical service failures prominently (e.g., query-scheduler, query-worker down)

🤖 Prompt for AI agents

Add health status visualization to the WebUI. This phase implements the frontend
display of service health information.

Implement WebUI server health API:
- Create route file for GET /api/health that proxies to API server's GET /health
endpoint
- Handle connection errors gracefully
- Transform response if needed for WebUI format

Create React health status hook:
- Create useHealthStatus.ts using React Query's useQuery
- Configure refetchInterval: 5000 for automatic polling
- Define TypeScript interfaces for health status response
- Return loading, error, and data states

Create health status display component:
- Use Ant Design's Tag, Badge, or Alert components
- Map health states to colors: green (healthy), red (unhealthy), gray
(unknown/stale)
- Display service name, status, and error message
- Support compact and detailed display modes
- Make responsive and accessible

Integrate into UI:
- Update App.tsx or main layout to include health status
- Add health status indicator to header or navigation
- Consider dedicated health status page/modal
- Use useHealthStatus hook
- Show loading state during initial fetch
- Display critical failures prominently

Phase 4: Deployment Configuration

This phase updates deployment configurations to enable health reporting by adding necessary environment variables and configuration values to Docker Compose and Helm charts.

Task 1: Update Docker Compose Configuration

Add health reporting configuration to docker-compose deployment files.

• Update tools/deployment/package/docker-compose-all.yaml to add environment variables to each service:
  • CLP_HEALTH_REPORTING_ENABLED=true
  • CLP_HEALTH_REPORTING_API_URL=http://api_server:3001
  • CLP_HEALTH_REPORTING_INTERVAL=10
• Add these variables to query-scheduler, compression-scheduler, query-worker, compression-worker, reducer, and garbage-collector service definitions
• Use YAML anchors (x-health-reporting-defaults) to reduce duplication following existing patterns
• Ensure API server starts before other services using depends_on (already present)

Task 2: Update Helm Chart Configuration

Add health reporting configuration to Kubernetes Helm chart values.

• Update tools/deployment/package-helm/values.yaml to include health_reporting section under each service in clpConfig
• Set default values: enabled: true, api_server_url: "http://api-server:3001", interval_seconds: 10
• Update deployment templates in tools/deployment/package-helm/templates/ to pass these values as environment variables to containers
• Use Helm's templating to reference the values (e.g., {{ .Values.clpConfig.query_scheduler.health_reporting.enabled }})
• Follow existing patterns for environment variable injection from ConfigMap or directly in deployment specs

🤖 Prompt for AI agents

Update deployment configurations to enable health reporting. This phase adds
necessary environment variables and configuration values.

Update Docker Compose:
- Add environment variables to docker-compose-all.yaml:
CLP_HEALTH_REPORTING_ENABLED, CLP_HEALTH_REPORTING_API_URL,
CLP_HEALTH_REPORTING_INTERVAL
- Add to query-scheduler, compression-scheduler, query-worker,
compression-worker, reducer, and garbage-collector services
- Use YAML anchors (x-health-reporting-defaults) to reduce duplication

Update Helm Chart:
- Add health_reporting section under each service in values.yaml with enabled,
api_server_url, and interval_seconds
- Update deployment templates to pass values as environment variables using Helm
templating
- Follow existing patterns for environment variable injection

---

🚀 Next Steps

🤖 All AI agent prompts combined

Task: 1

Establish the core health monitoring infrastructure in the API server. This
phase creates the foundation for tracking service health states.

Extend the Pydantic configuration schema in clp-py-utils:
- Create HealthReporting model with enabled, api_server_url, interval_seconds,
and timeout_seconds fields
- Add health_reporting field to QueryScheduler and CompressionScheduler models
- Create Reducer and GarbageCollector models with health_reporting field

Implement API server health cache:
- Create health_manager.rs module with HealthManager struct
- Use thread-safe cache (Arc<DashMap> or Arc<RwLock<HashMap>>)
- Define HealthStatus struct with service_name, instance_id, last_heartbeat,
status, error_message, and status_details
- Implement update_heartbeat(), get_service_health(), get_all_health(), and
mark_stale_as_unhealthy() methods
- Add HealthManager to Client state struct

Add health endpoints:
- POST /health endpoint accepting service_name, instance_id, timestamp,
error_message, and status_details
- GET /health endpoint returning all service health statuses
- GET /health with query parameters for specific service/instance
- Use Utoipa macros for OpenAPI documentation
- Return appropriate status codes (200 for healthy, 503 for unhealthy)

Add background cleanup task:
- Create Tokio task running every 10 seconds to mark stale services as unhealthy
- Use 30-second threshold for stale detection
- Handle graceful shutdown
===============================================================================

Task: 2

Add heartbeat reporting to all Python services. This phase implements periodic
health status reporting from services to the API server.

Create shared health reporter utility:
- Implement AsyncHealthReporter class using aiohttp with start(), stop(),
report_healthy(), and report_unhealthy() methods
- Implement SyncHealthReporter class using requests and threading.Thread
(daemon=True)
- Accept API URL, service name, interval, and timeout in constructor
- Use HOSTNAME environment variable for instance_id with fallback to
socket.gethostname()
- Handle connection failures gracefully

Integrate health reporting into services:
- Query Scheduler: Initialize AsyncHealthReporter in main(), create asyncio
task, include in asyncio.wait() with FIRST_COMPLETED, handle shutdown
- Compression Scheduler: Initialize SyncHealthReporter, call start() before main
loop, call stop() in cleanup
- Garbage Collector: Initialize AsyncHealthReporter in main(), add to task list
monitored by asyncio.wait()
- Reducer Service: Initialize separate AsyncHealthReporter instance with service
name "reducer", create task alongside reducer handler
- Celery Workers: Create health_reporter.py module, use
@signals.worker_ready.connect and @signals.worker_shutdown.connect, initialize
in query and compress celery.py files

Make all health reporting optional based on configuration
===============================================================================

Task: 3

Add health status visualization to the WebUI. This phase implements the frontend
display of service health information.

Implement WebUI server health API:
- Create route file for GET /api/health that proxies to API server's GET /health
endpoint
- Handle connection errors gracefully
- Transform response if needed for WebUI format

Create React health status hook:
- Create useHealthStatus.ts using React Query's useQuery
- Configure refetchInterval: 5000 for automatic polling
- Define TypeScript interfaces for health status response
- Return loading, error, and data states

Create health status display component:
- Use Ant Design's Tag, Badge, or Alert components
- Map health states to colors: green (healthy), red (unhealthy), gray
(unknown/stale)
- Display service name, status, and error message
- Support compact and detailed display modes
- Make responsive and accessible

Integrate into UI:
- Update App.tsx or main layout to include health status
- Add health status indicator to header or navigation
- Consider dedicated health status page/modal
- Use useHealthStatus hook
- Show loading state during initial fetch
- Display critical failures prominently
===============================================================================

Task: 4

Update deployment configurations to enable health reporting. This phase adds
necessary environment variables and configuration values.

Update Docker Compose:
- Add environment variables to docker-compose-all.yaml:
CLP_HEALTH_REPORTING_ENABLED, CLP_HEALTH_REPORTING_API_URL,
CLP_HEALTH_REPORTING_INTERVAL
- Add to query-scheduler, compression-scheduler, query-worker,
compression-worker, reducer, and garbage-collector services
- Use YAML anchors (x-health-reporting-defaults) to reduce duplication

Update Helm Chart:
- Add health_reporting section under each service in values.yaml with enabled,
api_server_url, and interval_seconds
- Update deployment templates to pass values as environment variables using Helm
templating
- Follow existing patterns for environment variable injection

💡 Iterate on the plan with: @coderabbitai <feedback>

Example Feedback
- @coderabbitai You can skip phase 3. Add a simple unit test case for phase 2.
- @coderabbitai For assumption 1 go ahead with option 3 and replan.

💬 Have feedback or questions? Drop into our discord or schedule a call!