Project Status: The basic CRD has been created. The metrics extraction and testing is ongoing right now...
Celery Operator deploys a quick celery solution within Kubernetes. You can manage multiple celery clusters within the Kubernetes through this operator. This operator is powered by operator framework. So, this is a more native way to create the celery infrastructure.
The tetative features supported are listed here.
- Dependency Free - No external broker is needed. It will spin up a broker for you automatically
- Mutiple Worker Pools - You can configure different worker pools for different queue
- Built-in HPA supported - A simple autoscaler based on resource usage
Here is the tracker for the MVP of this operators. After MVP, alpha version will be implemented.
- Pod Spawning
- Redis Deployment Broker
- Scheduler Deployment
- Worker
- Pod Scaling
- Worker
- Scheduler
- Field update on the CRD
- Worker
- Scheduler
- Broker
- Celery Stack
- HPA
- Implement scaling in broker
- Implement scaling in worker
- Implement scaling in scheduler
- Metric Export
- Get the metric from Broker
- Display the metric in Get and Describe API
- Create Metric Endpoint
- Testing
- Basic Celery Object Creation
- Pod delete and respawning testing
- Task Queuing
Please note that this has not been production ready yet.
These are the tech blogs to record the development process.