WIP - feat(deployment): Add k8s Helm chart for CLP package deployment.

docs/src/dev-docs/design-deployment-orchestration.md

@@ -1,8 +1,9 @@
 # Deployment orchestration
 
-The CLP package is composed of several components that are currently designed to be deployed in a
-set of containers that are orchestrated using a framework like [Docker Compose][docker-compose].
-This document explains the architecture of that orchestration and any associated nuances.
+The CLP package is composed of several components that are designed to be deployed in a set of
+containers that are orchestrated using a framework like [Docker Compose][docker-compose] or
+[Kubernetes][kubernetes] (via [Helm][helm]). This document explains the architecture of that
+orchestration and any associated nuances.
 
 ## Architecture
 
@@ -188,102 +189,96 @@ graph LR
 **Table 2**: One-time initialization jobs in the CLP package.
 ::::
 
-## Code structure
+## Orchestration methods
 
-The orchestration code is split up into:
+CLP supports two orchestration methods: Docker Compose for single-host or manual multi-host
+deployments, and Helm for Kubernetes deployments. Both methods share the same configuration
+interface (`clp-config.yaml` and `credentials.yaml`) and support the same deployment types.
 
-* `BaseController` that defines:
-  * common logic for preparing the environment variables, configuration files, and directories
-    necessary for each service.
-  * abstract methods that orchestrator-specific derived classes must implement in order to
-    orchestrate a deployment.
-* `<Orchestrator>Controller` that implements (and/or overrides) any of the methods in
-  `BaseController` (`<Orchestrator>` is a placeholder for the specific orchestrator for which the
-  class is being implemented).
+### Configuration
 
-## Docker Compose orchestration
+Each service requires configuration values passed through config files, environment variables,
+and/or command line arguments. Since services run in containers, some values must be adapted for the
+orchestration environment—specifically, host paths must be converted to container paths, and
+hostnames/ports must use service discovery mechanisms.
 
-This section explains how we use Docker Compose to orchestrate the CLP package and is broken into
-the following subsections:
+The orchestration controller (e.g., `DockerComposeController`) reads `etc/clp-config.yaml` and
+`etc/credentials.yaml`, then generates:
+* A container-specific CLP config file with adapted paths and service names
+* Runtime configuration (environment variables or ConfigMaps)
+* Required directories (e.g., data output directories)
 
-* [Setting up the Docker Compose project's environment](#setting-up-the-environment)
-* [Starting and stoping the Docker Compose project](#starting-and-stopping-the-project)
-* [Deployment types](#deployment-types)
-* [Implementation details](#implementation-details)
-* [Troubleshooting](#troubleshooting)
+For Docker Compose, this generates `var/log/.clp-config.yaml` and `.env`. For Kubernetes, the Helm
+chart generates a ConfigMap and Secrets from `values.yaml`.
 
-### Setting up the environment
-
-Several services require configuration values to be passed in through the CLP package's config file,
-environment variables, and/or command line arguments. Since the services are running in containers,
-some of these configuration values need to be modified for the orchestration environment.
-Specifically:
+:::{note}
+A `KubernetesController` is also planned that will read `clp-config.yaml` and `credentials.yaml`
+like `DockerComposeController`, then set up the Helm release accordingly. This will unify the
+configuration experience across both orchestration methods.
+:::
 
-1. Paths on the host must be converted to appropriate paths in the container.
-2. Component hostnames must be converted to service names, and component ports must be converted to the component's default ports.
-    * This ensures that in the Docker Compose configuration, services can communicate over fixed, predictable hostnames and ports rather than relying on configurable variables.
+### Secrets
 
-To achieve this, before starting the deployment, `DockerComposeController.start` generates:
+Sensitive credentials (database passwords, API keys) are stored in `etc/credentials.yaml` and
+require special handling to avoid exposure.
 
-* a CLP configuration file (`<clp-package>/var/log/.clp-config.yaml` on the host) specific to the
-  Docker Compose project environment.
-* an environment variable file (`<clp-package>/.env`) for any other configuration values.
-* any necessary directories (e.g., data output directories).
+* **Docker Compose**: Credentials are written to `.env` and passed as environment variables
+* **Kubernetes**: Credentials are stored in Kubernetes Secrets
 
-The Docker Compose project then passes those environment variables to the relevant services, either
-as environment variables or command line arguments, as necessary.
+### Dependencies
 
-### Starting and stopping the project
+As shown in [Figure 1](#figure-1), services have complex interdependencies. Both orchestrators
+ensure services start only after their dependencies are healthy.
 
-To start and stop the project, `DockerComposeController` simply invokes `docker compose up` or
-`docker compose down` as appropriate. However, to allow multiple CLP packages to be run on the same
-host, we explicitly specify a project name for the project, where the name is based on the package's
-instance ID.
+* **Docker Compose**: Uses `depends_on` with `condition: service_healthy` and container healthchecks
+* **Kubernetes**: Uses init containers (via the `clp.waitFor` helper) and readiness/liveness probes
 
-### Deployment Types
+### Storage
 
-CLP supports four deployment types determined by the `compression_scheduler.type` and
-`package.query_engine` configuration setting.
+Services require persistent storage for logs, data, archives, and streams.
 
-| Deployment Type | Compression Scheduler | Query Engine                 | Docker Compose File                |
-|-----------------|-----------------------|------------------------------|------------------------------------|
-| Base            | Celery                | [Presto][presto-integration] | `docker-compose-base.yaml`         |
-| Full            | Celery                | Native                       | `docker-compose.yaml`              |
-| Spider Base     | Spider                | [Presto][presto-integration] | `docker-compose-spider-base.yaml`  |
-| Spider Full     | Spider                | Native                       | `docker-compose-spider.yaml`       |
+* **Docker Compose**: Uses bind mounts for host directories and named volumes for database data.
+  Conditional mounts use variable interpolation to mount empty tmpfs when not needed.
+* **Kubernetes**: Uses PersistentVolumeClaims per component, with shared PVCs (`ReadWriteMany`) for
+  archives and streams. Uses `local-storage` StorageClass by default.
 
-### Implementation details
+### Deployment types
 
-One notable implementation detail is in how we handle mounts that are only necessary under certain
-configurations. For instance, the input logs mount is only necessary when the `logs_input.type` is
-`fs`. If `logs_input.type` is `s3`, we shouldn't mount some random directory from the user's
-host filesystem into the container. However, Docker doesn't provide a mechanism to perform
-conditional mounts. Instead, we use Docker's variable interpolation to conditionally mount an empty
-tmpfs mount into the container. This strategy is used wherever we need a conditional mount.
+CLP supports multiple deployment configurations based on the compression scheduler and query engine.
 
-### Troubleshooting
+| Deployment Type | Compression Scheduler | Query Engine                 |
+|-----------------|-----------------------|------------------------------|
+| Base            | Celery                | [Presto][presto-integration] |
+| Full            | Celery                | Native                       |
+| Spider Base     | Spider                | [Presto][presto-integration] |
+| Spider Full     | Spider                | Native                       |
 
-If you encounter issues with the Docker Compose deployment, first determine the instance ID for your
-deployment by checking the content of `<clp-package>/var/log/instance-id`. Then run one of the
-commands below as necessary.
+:::{note}
+Spider support is not yet available for Helm.
+:::
 
-1. Check service status:
+Docker Compose selects the appropriate compose file (e.g., `docker-compose.yaml` for Full,
+`docker-compose-spider.yaml` for Spider Full) and uses `deploy.replicas` with environment
+variables (e.g., `CLP_MCP_SERVER_ENABLED`) to toggle optional services. Helm uses conditional
+templating to include/exclude resources.
 
-   ```bash
-   docker compose --project-name clp-package-<instance-id> ps
-   ```
+## Troubleshooting
 
-2. View service logs:
+When issues arise, use the appropriate commands for your orchestration method:
 
-   ```bash
-   docker compose --project-name clp-package-<instance-id> logs <service-name>
-   ```
+* [Docker Compose debugging][docker-compose-debugging]
+* [Kubernetes Helm debugging][kubernetes-debugging]
 
-3. Validate configuration:
+## User guides
 
-   ```bash
-   docker compose config
-   ```
+* [Kubernetes deployment][kubernetes-guide]: Deploying CLP with Helm
+* [Multi-host deployment][docker-compose-multi-host]: Manual Docker Compose across multiple hosts
 
 [docker-compose]: https://docs.docker.com/compose/
+[docker-compose-debugging]: ../user-docs/guides-docker-compose-deployment.md#monitoring-and-debugging
+[helm]: https://helm.sh/
+[kubernetes]: https://kubernetes.io/
+[kubernetes-debugging]: ../user-docs/guides-k8s-deployment.md#monitoring-and-debugging
+[kubernetes-guide]: ../user-docs/guides-k8s-deployment.md
+[docker-compose-multi-host]: ../user-docs/guides-docker-compose-deployment.md#multi-host-deployment
 [presto-integration]: ../user-docs/guides-using-presto.md

docs/src/user-docs/guides-multi-host.md → docs/src/user-docs/guides-docker-compose-deployment.md

@@ -1,12 +1,28 @@
-# Multi-host deployment
+# Docker Compose deployment
+
+This guide explains how to deploy CLP using Docker Compose. Docker Compose provides a
+straightforward way to orchestrate CLP's services, suitable for both development and production
+environments.
+
+## Deployment options
+
+Docker Compose can be used for:
+
+* **Single-host deployment**: Run all CLP services on a single machine. This is the simplest setup,
+  covered in the [quick-start guides](quick-start/index.md).
+* **Multi-host deployment**: Distribute CLP services across multiple machines for higher throughput
+  and scalability. This is covered in detail below.
+
+---
+
+## Multi-host deployment
 
 A multi-host deployment allows you to run CLP across a distributed set of hosts.
 
-:::{warning}
-The instructions below provide a temporary solution for multi-host deployment and may change as we
-actively work to improve ease of deployment. The present solution uses *manual* Docker Compose
-orchestration; however, Kubernetes Helm support will be available in a future release, which will
-simplify multi-host deployments significantly.
+:::{note}
+The instructions below use *manual* Docker Compose orchestration, which is more lightweight and
+provides fine-grained control over service placement, but requires more configuration than
+Helm-based deployments.
 :::
 
 ## Requirements
@@ -305,9 +321,11 @@ sbin/stop-clp.sh
 
 This will stop all CLP services managed by Docker Compose on the current host.
 
-## Monitoring services
+## Monitoring and debugging
 
-To check the status of services on a host:
+First, determine your instance ID from `<clp-package>/var/log/instance-id`.
+
+To check the status of services:
 
 ```bash
 docker compose --project-name clp-package-<instance-id> ps
@@ -319,6 +337,20 @@ To view logs for a specific service:
 docker compose --project-name clp-package-<instance-id> logs -f <service-name>
 ```
 
+To execute commands in a running container:
+
+```bash
+docker compose --project-name clp-package-<instance-id> exec <service-name> /bin/bash
+```
+
+To validate your Docker Compose configuration:
+
+```bash
+docker compose config
+```
+
+---
+
 ## Setting up SeaweedFS
 
 The instructions below are for running a simple SeaweedFS cluster on a set of hosts. For other use

docs/src/user-docs/guides-external-database.md

@@ -205,7 +205,7 @@ initialization jobs (`db-table-creator` and `results-cache-indices-creator`).
 
 [aws-rds]: https://aws.amazon.com/rds/
 [azure-databases]: https://azure.microsoft.com/en-us/products/category/databases
-[docker-compose-orchestration]: ../dev-docs/design-deployment-orchestration.md#docker-compose-orchestration
+[docker-compose-orchestration]: ../user-docs/guides-docker-compose-deployment.md
 [mongodb-install]: https://www.mongodb.com/docs/manual/installation/
 [mongodb-security]: https://docs.mongodb.com/manual/security/
-[multi-host-guide]: guides-multi-host.md#starting-clp
+[multi-host-guide]: guides-docker-compose-deployment.md#starting-clp