Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
44 changes: 13 additions & 31 deletions docs/inkless/ts-unification/DESIGN.md
Original file line number Diff line number Diff line change
Expand Up @@ -555,8 +555,14 @@ public class DisklessMigrationHandler {

### 4.4 Stream 4: Multi-Replica Model for Diskless Topics

> **📄 Detailed Design:** See [DISKLESS_MANAGED_RF.md](DISKLESS_MANAGED_RF.md) for the comprehensive design of rack-aware,
> KRaft-managed replicas for diskless topics, including approach comparison, cost analysis, and implementation path.

**Objective:** Enable diskless topics to use 3 actual Kafka replicas while preserving write-to-any semantics.

**Activation:** Controlled by `diskless.managed.rf.enabled` controller config (default: `false`). This allows incremental
rollout — when enabled, new diskless topics get RF=rack_count; existing topics are unaffected.

#### 4.4.1 Current vs. Proposed Model

**Current Model:**
Expand Down Expand Up @@ -618,37 +624,12 @@ public class AppendHandler {

#### 4.4.4 Metadata Transformer Updates

```java
// storage/inkless/src/main/java/io/aiven/inkless/metadata/InklessTopicMetadataTransformer.java
public class InklessTopicMetadataTransformer {

public void transformClusterMetadata(
final String clientId,
final Iterable<MetadataResponseData.MetadataResponseTopic> topicMetadata
) {
for (final var topic : topicMetadata) {
if (!metadataView.isDisklessTopic(topic.name())) {
continue;
}

for (final var partition : topic.partitions()) {
if (metadataView.isHybridTopic(topic.name())) {
// Hybrid topic: use actual replica assignment for RLM compatibility
// Leader selection based on partition metadata
// No transformation needed - use real replicas
} else {
// Pure diskless: current behavior - route to any broker
int leader = selectLeaderForInklessPartitions(clientId, topic.topicId(),
partition.partitionIndex());
partition.setLeaderId(leader);
partition.setReplicaNodes(List.of(leader));
partition.setIsrNodes(List.of(leader));
}
}
}
}
}
```
> **📄 See [DISKLESS_MANAGED_RF.md](DISKLESS_MANAGED_RF.md)** for the complete routing algorithm, including AZ-priority
> logic, mode derivation, and fallback behavior.

**Summary:** The transformer derives routing mode from topic config (`remote.storage.enable` + `diskless.enable`) and
routes requests with AZ-priority. Both modes prefer assigned replicas first; `DISKLESS_ONLY` can fall back to any broker
when replicas are unavailable, while `DISKLESS_TIERED` stays on replicas only (RLM requires `UnifiedLog` state).

### 4.5 Stream 5: RLM Integration for Hybrid Topics

Expand Down Expand Up @@ -1972,6 +1953,7 @@ Once the tiering pipeline is complete, enabling reverse migration becomes straig

- [KIP-405: Tiered Storage](https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage)
- [Inkless Architecture Documentation](./architecture.md)
- [Diskless-Managed Replication Factor](DISKLESS_MANAGED_RF.md) — Detailed design for rack-aware, KRaft-managed replicas
- [RemoteLogManager Implementation](../../core/src/main/java/kafka/log/remote/RemoteLogManager.java)
- [ControlPlane Interface](../../storage/inkless/src/main/java/io/aiven/inkless/control_plane/ControlPlane.java)

Loading