[RoadMap][Call For Contributions] Mooncake Store V3 Roadmap

[stmatengss]

Milestone 1: Core Architecture Refactor & Decoupling

This milestone focuses on foundational architectural changes to improve modularity, flexibility, and prepare for future scaling.

• (TE/Store Separation): Decouple the TE (Task/Tensor Engine) and Store components into separate, independent packages.
• (Client/Worker Decoupling): Decouple the dummy client from the worker to remove strong dependencies. @YiXR
  • feat[Store]: Add standalone deployment implementation for Client #1084
  • feat[Store]: Add multi dummy clients support for real client #1122
  • feat[Store]: Refine shm mmap logic and add reconnection for Dummy Client after the Real restarted #1146
  • feat[Store]: Introduce shm helper for dummy #1177
  • feat[Store]: add multi shm support for dummy and real client #1206
• (Flexible Deployment): Update the Store to support various flexible deployment models, such as client-only, client + master, etc. 
• (Tensor-native APIs): Put/Get Tensor APIs contain TP rank and model info.
  • [Store] add tp awareness for get_tensor #1127
• (Multiple Replicas): Support a replica list for each KV item.
  • [Store] pub_tensor for multiple replica #1148
• (Layer-wise Storage): Maintaining layer info on the Mooncake side, supports stream transfer and communication overlap.

Milestone 2: Master Service Enhancements

This milestone enhances the Master component to support new storage architectures and routing logic.

• (Key-based Routing): Implement new key-based routing capabilities in the Master service. 
• (Metadata Adaptation - Storage): Adapt the Master's metadata management to support the new multi-level storage architecture.
• (Recovery) kv metadata persistency [Store]wrapper etcd basic interface for master service HA #1451
• (KVCache Awareness Interface) Exposes hit ratio for different layers.
• (Metadata Adaptation - HA): Upgrade metadata schema and logic to meet new High Availability (HA) requirements. 
  • Snapshot
    • [RFC]: [Mooncake - Master] Persistence and Recovery Solution for KV Metadata and Segment Information #1150
    • [Store]: add ETCD snapshot object store backend in HA mode #1381
    • [Store] Implement Metadata Persistence and Recovery for Master Service #1431
    • [Store] add serializer and deserializer in task manager for snapshot #1465
  • Oplog
    • [RFC]: Hot Standby Mode for Master Service Metadata High Availability #1200
    • [Store]wrapper etcd basic interface for master service HA #1451
    • [Store][Feature]Hot Standby and Oplog Interface for Master Service HA #1515
  • ...
• (Multi-tenant): Support Multi-tenant with different models, users and auth keys

Milestone 3: Worker: Multi-Level Storage Architecture

This is a major epic to build the next-generation multi-level storage system within the Worker.

• 3.1: Abstraction & Caching

  • (Storage Abstraction Layer): Design and implement the core abstraction layer for multi-level storage. 
  • (Cache Scheduling Interface): Design the abstract interface for cache scheduling logic. 
  • (Eviction Logic): Implement basic data eviction logic within the new storage architecture.
    • @Vincent-Bo-ali [store] Add disk eviction feature #1028
  • (LRU Cache): Implement an LRU (Least Recently Used) policy as the default cache scheduling strategy. 
  • (Local Client Cache): Keep a local cache for better performance.
    • [RFC]: Add Local Cache Mechanism for Mooncake Store Client #1062

• 3.2: Storage Backend Implementation

  • (DRAM Adaptation): Adapt the storage layer for DRAM, including support for NUMA affinity. 
  • (SSD Adaptation): Adapt the storage layer for SSDs, enabling local external storage read/write capabilities. 
    • [Store] feat: store key-value data in buckets #968
    • [Store] feat: Implement a FileStorage component to manage the lifecycle of key-value data #1031
    • [RFC] Contribute Local Storage to Distributed Pool #1054
    • feat(Store): Support get local ssd object #1203
    • [Store]: Add OffsetAllocator disk backend with lock-striped metadata + refcounted extents. #1284
    • [Store] - Implement partial success handling in BatchOffload #1319
    • [Store] - Optimize BucketStorageBackend for reduced lock contention and add delete safety #1456
  • (VRAM Adaptation): Adapt the storage layer to utilize VRAM. 
  • (Huawei NPU Adaptation): Implement support for Huawei NPUs (H2D). 
    • @ascend-direct-dev

• 3.3: Elastic KVCache Storage

  • (KVCache Migration): Move KVCache between Mooncake client.
    • [Store][Feature]: add task manager component #1326
  • (Data Replica): Dynamic Replication
    • [RFC]: dynamic replica management #1100

Milestone 4: Worker: Networking & Elasticity

This milestone focuses on refactoring worker communication and enabling resource elasticity.

• (RPC Refactor): [Phase 1] Refactor the worker's read/write logic to replace RDMA with RPC-based communication. 
• (Barex Transport Support): Support Alibaba barex transport in TE for Mooncake Store.
  • feat[accl-barex]: add barex_transport by build with USE_BAREX #1045
• (Resource Elasticity): Implement single-worker resource elasticity. 
• (Event‑driven completion): Provide an option to using event-driven notification worker instead of busy-polling.
  • [Performance]: High CPU usage due to busy-wait in TransferEngineOperationState::wait_for_completion #1033
  • [Store|TransferEngine]: use condition-variable based completion instead of busy-polling #1053
• (IPv6 Support): Support IPv6 in client, master and metadata server.
  • [Bug]: Error in IPv6 support for mooncake 0.3.6 on vLLM 0.11.0 #1043
  • TCP transport support ipv6 #1067

Milestone 5: Deployment & Operations

This milestone covers K8s integration (i.e., RBG, https://github.com/sgl-project/rbg) and build process improvements.

• (K8s Autoscaling): Implement support for Kubernetes-based autoscaling of worker and dummy client instances.
• (Scenario-based Builds): Implement a build system capable of producing different worker binaries optimized for different scenarios. 
• (Integration With AI Configurator): Use AI Configurator for better measuring Resource workers and other configurations.
• (Deployment Documentation & Guides): Create comprehensive, up-to-date deployment documentation and step-by-step setup guides to simplify installation and configuration for all environments.

Milestone 6: Conductor

This feature implements a standalone or co-located conductor for use by the Gateway. @zhongzhouTan-coder @yejj710 @Liziqi-77 @Keithwwa @Asher-XunZhang

• (Basic design) : Discuss and design the standard specifications for KV indexer and kv-events
  • [RFC]: Mooncake KV-Store Indexer API Standardization #1403
  • [RFC]: Kv Events API Standardization #1527
• (KV indexer): Implement a Mooncake KV-Store Indexer service (include /quey, /query_by_hash ..etc)
• (KV publisher): support kv-event publisher in Mooncake store
• (Service registration): support two methods for service registration: either through static configuration files or dynamic service registration APIs
• (KV events): Support consuming KV events generated by inference frameworks and Mooncake

Milestone 7: Hardware Compatibility

• (AMD) HIP, ROCm, MORI
  • Improve AMD HIP support with hipify-perl #1154
  • [TE/Examples] Memory initialization and HIP cleanup fixes #1179
  • [NvlinkTransport] Fix HIP Shareable POSIX File Descriptor Handles #1189
• (Moore Thread)
  • Support AMDGPU (refactor CUDA-alike) #973
  • [MUSA] Enable USE_MNNVL #1176
• (AWS EFA)
  • [TE] Add AWS EFA transport using libfabric #1509
  • [TE] Support TCP fallback in EFA build and improve EFA documentation #1523
  • add efa protocol to mooncake store client #1526

Milestone 8: Pytorch Eco-system

• (Support Tensor Attributions)
• (Native torch format offload)

Milestone 9: CI & CD enhancement

• (End-to-end CI tests): For SGLang, support Hicache, PD, Elatics EP, checkpoint engine tests.
  • SGLang PD E2E CI: [CI] add sglang integration test #1089
  • Mooncake EP CI: Add a CI test for Mooncake EP Backend (CPU only) #1099
  • SGLang HiCache E2E CI: [CI] Add sglang e2e tests #1181
  • vLLM 1P1D E2E CI: [CI]Add vllm 1p1d test case #1491

Milestone 10: Performance & Benchmarks

• (Store Master Benchmark): Design and integrate a dedicated benchmark for the Mooncake store master module to evaluate throughput, latency, and scalability. [Store]: Add master_bench for benchmarking QPS of MasterService #1201
• (SSD/NVMe storage benchmark) : [Store] - Add comprehensive storage backend benchmark suite #1388

Milestone 11: RL Infra

• (ROLL):
• (Slime/Miles) :
  • Add Mooncake Backend for Rollout Data Transfer THUDM/slime#1709
  • [feat]support rollout data transfer with mooncake radixark/miles#591

---

Thanks for being a part of the Mooncake community! Welcome to discuss and contribute!

---

If you have any ideas, just leave a comment below and help shape the Roadmap.

[Keithwwa]

Celebrate! Finally, we have the V3 Roadmap!
I have a few questions I'd like to ask:

1. What is the motivation of 'key-based routing'?
2. Have you considered more diversified cache scheduling strategies, or designed scalability for it?
3. In addition, could you share some details about the 'Cache Scheduling Interface' and its design concepts?

[stmatengss]

Celebrate! Finally, we have the V3 Roadmap! I have a few questions I'd like to ask:

1. What is the motivation of 'key-based routing'?
2. Have you considered more diversified cache scheduling strategies, or designed scalability for it?
3. In addition, could you share some details about the 'Cache Scheduling Interface' and its design concepts?

1. It means we can provide KVCache location query service for Router.
2. No updates now; you can give any suggestions, and I will add them to the Roadmap.
3. It includes migration, promotion/demotion, and hot-data detection (hot-data will have more replicas).

[zhongzhouTan-coder]

Congratulations, just a big v3 roadmap!!!!
I am very interesting in the Cache Scheduling such as hot data migration, I hope I can join the contribution and do some work to help build a more powerful project~ ╰(°▽°)╯

[stmatengss]

Congratulations, just a big v3 roadmap!!!! I am very interesting in the Cache Scheduling such as hot data migration, I hope I can join the contribution and do some work to help build a more powerful project~ ╰(°▽°)╯

Cool! It's a important feature for elastic deployment. Welcom to join the slack channel for offline discussion.

[stmatengss]

Mooncake Omni & Disaggregation

flowchart TB
    subgraph Client ["Client / Request"]
        Req["Multimodal Request"]
    end

    subgraph Inference_Arch ["Upper-Level Inference Architecture (EPD)"]
        direction TB
        Router["Router / Scheduler"]
        
        subgraph Compute_Nodes ["Compute Nodes"]
            Encoder["Encoder Nodes<br/>(Multimodal Processing)"]
            Prefill["Prefill Nodes<br/>(Context Processing)"]
            Decode["Decode Nodes<br/>(Token Generation)"]
        end
        
        Tokenizer["Tokenizer +<br/>MM Pre-processor"]
    end

    subgraph Mooncake_System ["Mooncake Disaggregated System"]
        direction TB
        MC_Store[("Mooncake Store /<br/>Transfer Engine")]
        
        %% Logical Data Stores within Mooncake
        subgraph Data_Types ["Data Managed"]
            KV_Store[("1. KV Cache Storage")]
            Emb_Store[("4. Embedding Data Storage")]
            Rollout_Store[("2. RL Rollout Data Storage")]
            Weight_Store[("3. Model Weights Storage<br/>(Checkpoint Engine)")]
        end
    end

    subgraph RL_Train_Frame ["RL & Training Framework"]
        direction TB
        Rollout_Worker["Rollout Workers<br/>(Inference Actors)"]
        Trainer["Training Workers<br/>(Learners)"]
    end

    %% Inference Flow
    Req --> Tokenizer
    Tokenizer --> Router
    Router --> Encoder
    
    Encoder -- "Compute Embeddings" --> Emb_Store
    Encoder -.-> Prefill
    
    Emb_Store -- "Zero-Copy Transfer" --> Prefill
    Prefill -- "Generate & Store KV" --> KV_Store
    Prefill -- "Handover" --> Decode
    
    KV_Store <-- "Load/Offload KV" --> Decode

    %% RL & Training Flow
    Rollout_Worker -- "Generate Experience" --> Rollout_Store
    Rollout_Store -- "Fetch Batch" --> Trainer
    
    Trainer -- "Update Weights" --> Weight_Store
    Weight_Store -- "Distribute New Weights" --> Rollout_Worker
    Weight_Store -. "Update Serving Weights" .-> Compute_Nodes

    %% Styles
    style Mooncake_System fill:#f9f,stroke:#333,stroke-width:2px,color:black
    style Data_Types fill:#fff,stroke:#666,stroke-dasharray: 5 5,color:black
    style KV_Store fill:#ffdb99,stroke:#d48806
    style Emb_Store fill:#ffdb99,stroke:#d48806
    style Rollout_Store fill:#b5f5ec,stroke:#08979c
    style Weight_Store fill:#d9f7be,stroke:#389e0d

Loading