Add support for deploying CLP to Kubernetes clusters using Helm charts.

[junhaoliao]

Request

Currently, CLP package components can only be orchestrated by invoking docker subprocesses in components/clp-package-utils/clp_package_utils/scripts/start_clp.py.

A request to migrate the orchestration to Docker Compose is currently in progress: #1177 -> #1178

This request aims to extend the deployment capabilities to support Kubernetes, providing users with a more scalable and production-ready deployment option.

The goal is to provide Helm charts and a Kubernetes "deployment controller" that can provision and deploy all CLP components to a Kubernetes cluster, similar to how the Docker Compose implementation works.

Possible implementation

1. Helm charts for all components: Implement comprehensive Helm charts for all CLP components with proper configuration

   • Component charts:
     • Infrastructure services:
       • database (MySQL)
       • queue (RabbitMQ)
       • redis
       • results_cache (MongoDB)
     • Application services:
       • compression_scheduler
       • query_scheduler
       • compression_worker
       • query_worker
       • reducer
       • webui
       • garbage_collector
   • Chart features:
     • Persistent storage with PersistentVolumeClaims for data persistence
     • Configuration management using ConfigMaps for package configurations
     • Health checks with liveness and readiness probes for components
     • Resource management with CPU and memory limits/requests
     • Scaling support with configurable replica counts for worker components
     • Service discovery with proper service definitions for inter-component communication

2. Kubernetes "deployment controller" implementation: Implement a dedicated Kubernetes controller to handle deployment to Kubernetes clusters

   • Responsibilities:
     • Configuration translation to convert CLP configuration files to Kubernetes-compatible formats (Helm values)
     • Helm chart deployment to deploy CLP components using Helm charts
     • Namespace management to create and manage Kubernetes namespaces for CLP deployments
     • Platform abstraction to provide the same interface as DockerComposeController for consistent usage
   • Implementation:
     • The Kubernetes controller will inherit from the existing BaseController class, similar to DockerComposeController
     • It will be conditionally imported and used based on a --platform flag in the start/stop scripts
     • The controller will support a dry-run mode that generates Helm configuration values without actually starting the deployment, allowing users to tailor the configuration to their specific needs before deploying

3. Updated deployment scripts: Enhance the start/stop scripts to support Kubernetes deployment

   • start_clp.py enhancements:
     • Add --platform option with choices docker (default) and k8s
     • Conditionally import and use KubernetesController when platform is k8s
     • Maintain backward compatibility with existing Docker Compose deployments
   • stop_clp.py enhancements:
     • Add --platform option with choices docker (default) and k8s
     • Conditionally import and use KubernetesController when platform is k8s
     • Maintain backward compatibility with existing Docker Compose deployments