Create a design-document for the controller #181
Conversation
github-actions
bot
added
the
documentation
lllamnyp
changed the title
|
Could you please move to design subdirectory into website |
|
I am a bit confused with this section of flow: I would suggest a bit updated way to control resources in this case: |
|
Also, I didn't get the purpose of the steps:
I mean, could you please list cases that we avoid using these checks? |
|
Let's move it into |
This commit includes all changes for feature #181 that have not been split up into small stacked PRs. It should not be merged and will later be undone and split into smaller logical chunks of work.
…th check procedure (#252) This is the first PR in a series of [stacked PRs](https://www.stacking.dev/), aimed ultimately at implementing the features described in #181 and #207. The next PR in the stack can be found at #259.
This commit includes all changes for feature #181 that have not been split up into small stacked PRs. It should not be merged and will later be undone and split into smaller logical chunks of work.
This commit includes all changes for feature #181 that have not been split up into small stacked PRs. It should not be merged and will later be undone and split into smaller logical chunks of work.
WalkthroughThe pull request updates the Changes
Sequence DiagramsequenceDiagram
participant Operator as EtcdCluster Operator
participant Cluster as Kubernetes Cluster
participant Service as Etcd Service
participant StatefulSet as Etcd StatefulSet
Operator->>Cluster: Check Service Endpoints
alt Endpoints Exist
Operator->>Service: Connect and Fetch Status
Operator->>Cluster: Validate Cluster ID
Operator->>Cluster: Check Quorum Status
Operator->>Cluster: Manage Cluster Members
else No Endpoints
Operator->>Cluster: Check StatefulSet Existence
Operator->>Cluster: Create/Reconcile Configuration
end
Poem
Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media? 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (4)
docs/DESIGN.md (4)
1-5: Enhance the document introduction for better context and navigation.Consider expanding the introduction to include:
- Brief overview of what EtcdCluster is and its purpose
- Controller's key responsibilities and objectives
- Table of contents or document structure
- Prerequisites or assumptions
# Design This document describes the interaction between `EtcdCluster` custom resources and other Kubernetes primitives and gives an overview of the underlying implementation. + +## Table of Contents +- [Overview](#overview) +- [Prerequisites](#prerequisites) +- [Reconciliation flowchart](#reconciliation-flowchart) + +## Overview +EtcdCluster is a custom resource that manages etcd clusters in Kubernetes. This controller handles: +- Cluster state management +- Scaling operations +- Self-healing mechanisms +- Member addition and removal + +## Prerequisites +- Understanding of Kubernetes custom resources +- Familiarity with etcd cluster operations
6-7: Add context to the flowchart section.The flowchart would be more accessible with:
- Brief explanation of its purpose
- Description of key components and their relationships
- Legend explaining the symbols and color coding (if any)
## Reconciliation flowchart + +This flowchart illustrates the controller's reconciliation loop, showing how it: +- Manages cluster state transitions +- Handles error conditions +- Performs scaling operations + +### Legend +- Diamond shapes: Decision points +- Rectangular boxes: Actions +- Rounded boxes: Terminal states
50-56: Enhance error detection and recovery for unreachable members.The current flow for unreachable members could be more robust:
- No timeout specifications
- Limited diagnostics
- No progressive recovery steps
Consider adding more detailed diagnostic steps:
AAA --> |No members\nreached| AAAB{Is the STS\npresent?} AAAB --> |Yes| AAABA{"` - Does it have the correct pod spec? + Diagnostic checks: + 1. Pod spec validation + 2. Network connectivity + 3. DNS resolution + 4. Certificate validity + 5. Resource constraints `"}
8-73: Consider adding operational metrics and monitoring points.The flowchart would benefit from indicating key metrics collection points for operational visibility:
- Reconciliation loop duration
- Error rates by type
- State transition latencies
- Member health metrics
This would help operators monitor the controller's health and performance effectively. Consider adding annotations in the flowchart to indicate metric collection points or creating a separate section documenting the metrics strategy.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
⛔ Files ignored due to path filters (1)
docs/sts-flow.svgis excluded by!**/*.svg
📒 Files selected for processing (1)
docs/DESIGN.md(1 hunks)
🔇 Additional comments (1)
docs/DESIGN.md (1)
16-20: Clarify the "Promote any learners" step and its prerequisites.
As noted in the PR comments, the purpose of the "Promote any learners" step needs clarification:
- When and why are learners promoted?
- What conditions must be met before promotion?
- What happens if promotion fails?
Consider adding comments in the flowchart to explain these operations:
AAAAAA --> |Yes| AAAAAAA["`
- Promote any learners.
+ Promote learners to voting members when:
+ - Learner is fully synced
+ - Cluster has quorum
+ - No other promotions in progress
Ensure configmap with initial cluster matching existing members and cluster state=existing.
Ensure StatefulSet with replicas = max member ordinal + 1
`"]
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
There was a problem hiding this comment.
Actionable comments posted: 3
♻️ Duplicate comments (1)
docs/DESIGN.md (1)
36-54: 🛠️ Refactor suggestionEnhance error handling with automated recovery procedures.
Building upon the previous feedback about quorum loss, consider adding:
- Automated recovery procedures for common failure scenarios
- Integration with monitoring systems
- Clear escalation paths for non-recoverable errors
AAAAAAAA --> |No| AAAAAAAAB1[On timeout evict member.] - AAAAAAAAB1 --> AAAAAAAAB2[Delete PVC, ensure ConfigMap with\nmembers + this one and delete pod.] + AAAAAAAAB1 --> AAAAAAAAB0{Is error\nrecoverable?} + AAAAAAAAB0 --> |Yes| AAAAAAAAB2["` + 1. Record recovery attempt metrics + 2. Execute automated recovery: + - Delete PVC + - Update ConfigMap + - Delete pod + 3. Monitor recovery success + `"] + AAAAAAAAB0 --> |No| AAAAAAAAB3["` + 1. Raise alerts + 2. Record failure metrics + 3. Update status with error details + 4. Escalate to operators + `"]
🧹 Nitpick comments (2)
docs/DESIGN.md (2)
1-5: Enhance the introduction with more context and structure.Consider expanding the introduction to include:
- Background context about etcd and its role in Kubernetes
- Key concepts and terminology used throughout the document
- Overview of the document's structure and sections
# Design This document describes the interaction between `EtcdCluster` custom resources and other Kubernetes -primitives and gives an overview of the underlying implementation. +primitives and gives an overview of the underlying implementation. + +## Overview + +Etcd is a distributed key-value store that plays a crucial role in Kubernetes by storing cluster state. +This document outlines the design of a controller that manages etcd clusters within Kubernetes, +handling operations such as scaling, self-healing, and state management. + +### Key Concepts + +- **EtcdCluster**: Custom resource defining the desired state of an etcd cluster +- **StatefulSet**: Kubernetes primitive used to manage the etcd pods +- **Quorum**: Minimum number of healthy members required for cluster operations + +### Document Structure + +1. Reconciliation Flowchart: Detailed process flow for cluster management +2. State Transitions: Various cluster states and their handling +3. Error Scenarios: Recovery procedures and failure handling
1-81: Consider adding supplementary documentation sections.To make the design document more comprehensive, consider adding:
- High-level architecture overview
- Sequence diagrams for key operations (scaling, recovery)
- Examples of common scenarios and their handling
- Operational guidelines for administrators
Would you like assistance in generating these additional sections?
Motivation
I started some "R'n'D" (scare quotes intended) for implementing scale up, scale down, self-healing and so on and quickly realized, that the coding of the member add/member remove and similar steps is the more trivial part of the undertaking. The difficult part is coming up with a working algorithm that can correctly deduce the cluster's state and execute the necessary actions at the right time.
To better reason about the controller's algorithm now, and to better develop it going forward, I feel it is important to have good documentation of the current design and the intended next steps, so I started with trying to document the current state of the code.
Results
This document contains a mermaid flowchart that outlines the reconciliation loop. It is better viewed in rendered form.
Going forward, I envision this document to have at least three purposes:
Summary by CodeRabbit
EtcdClustercustom resources with a detailed flowchart illustrating the reconciliation process and lifecycle management within a Kubernetes environment.