[TransferEngine][feature] add the http metadata server. (#37)

* [TransferEngine][feature] add the http metadata server.

* add doc & golang http server demo.

Author: doujiang24

CMakeLists.txt

@@ -26,7 +26,8 @@ set(CMAKE_EXPORT_COMPILE_COMMANDS ON)
 
 option(USE_CUDA "option for using gpu direct" OFF)
 option(USE_CXL "option for using cxl protocol" OFF)
-option(USE_REDIS "option for enable redis support" OFF)
+option(USE_REDIS "option for enable redis as metadata server" OFF)
+option(USE_HTTP "option for enable http as metadata server" OFF)
 
 option(WITH_P2P_STORE "build p2p store library and sample code" OFF)
 option(WITH_RUST_EXAMPLE "build the Rust interface and sample code for the transfer engine" OFF)
@@ -39,7 +40,12 @@ endif()
 
 if (USE_REDIS)
   add_compile_definitions(USE_REDIS)
-  message(STATUS "Redis support is enabled")
+  message(STATUS "Redis as metadata server support is enabled")
+endif()
+
+if (USE_HTTP)
+  add_compile_definitions(USE_HTTP)
+  message(STATUS "Http as metadata server support is enabled")
 endif()
 
 add_subdirectory(mooncake-transfer-engine)

README.md

@@ -110,7 +110,8 @@ In addition, to support more features of Mooncake Transfer Engine, we *recommand
   ```
 - Go 1.20+, if you want to build with `-DWITH_P2P_STORE`. You may download it from [here](https://go.dev/dl/).
 - Rust Toolclain, if you want to build with `-DWITH_WITH_RUST_EXAMPLE`.
-- `hiredis`, if you want to build with `-DWITH_REDIS`, so that you use Redis instead of etcd as metadata servers.
+- `hiredis`, if you want to build with `-DUSE_REDIS`, so that you use Redis instead of etcd as metadata servers.
+- `curl`, if you want to build with `-DUSE_HTTP`, so that you use HTTP instead of etcd as metadata servers.
 
 ### Installation
 1. Init source code

dependencies.sh

@@ -32,6 +32,7 @@ sudo apt-get install -y build-essential \
                         libprotobuf-dev \
                         protobuf-compiler-grpc \
                         pybind11-dev \
+                        libcurl4-openssl-dev \
                         libhiredis-dev
 
 echo "*** Download and installing [cpprest sdk] ***"

doc/en/build.md

@@ -33,6 +33,7 @@ This document describes how to build Mooncake.
 - gcc: 10.2.1
 - go: 1.19+
 - hiredis
+- curl
 
 ### Steps
 
@@ -46,6 +47,7 @@ This document describes how to build Mooncake.
                    libgtest-dev \
                    libjsoncpp-dev \
                    libnuma-dev \
+                   libcurl4-openssl-dev \
                    libhiredis-dev
 
     # For centos/alibaba linux os
@@ -59,6 +61,8 @@ This document describes how to build Mooncake.
                 boost-devel \
                 openssl-devel \
                 protobuf-devel \
+                hiredis-devel \
+                libcurl-devel \
                 protobuf-compiler 
     ```
     NOTE: You may need to install gtest, glog, gflags from source code:
@@ -144,4 +148,5 @@ Mooncake supports the following advanced compile options:
 - `-DUSE_CXL=[ON|OFF]`: Enable CXL protocols. 
 - `-DWITH_P2P_STORE=[ON|OFF]`: Enable Golang support and build P2P Store. 
 - `-DWITH_WITH_RUST_EXAMPLE=[ON|OFF]`: Enable Rust language support.
-- `-DWITH_REDIS_=[ON|OFF]`: Enable Redis as metadata server in Mooncake (`hiredis` required).
+- `-DUSE_REDIS=[ON|OFF]`: Enable Redis as metadata server in Mooncake (`hiredis` required).
+- `-DUSE_HTTP=[ON|OFF]`: Enable Http as metadata server in Mooncake (`curl` required).

doc/en/transfer-engine.md

@@ -94,13 +94,28 @@ The sample program provided in `mooncake-transfer-engine/example/transfer_engine
 
 After successfully compiling Transfer Engine, the test program `transfer_engine_bench` can be found in the `build/mooncake-transfer-engine/example` directory.
 
-1. **Start the `etcd` service.** This service is used for the centralized highly available management of various metadata for Mooncake, including the internal connection status of Transfer Engine. It is necessary to ensure that both the initiator and target nodes can smoothly access this etcd service, so pay attention to:
-   - The listening IP of the etcd service should not be 127.0.0.1; it should be determined in conjunction with the network environment. In the experimental environment, 0.0.0.0 can be used. For example, the following command line can be used to start the required service:
+1. **Start the `metadata` service.** This service is used for the centralized highly available management of various metadata for Mooncake, including the internal connection status of Transfer Engine. It is necessary to ensure that both the initiator and target nodes can smoothly access this metadata service, so pay attention to:
+   - The listening IP of the metadata service should not be 127.0.0.1; it should be determined in conjunction with the network environment. In the experimental environment, 0.0.0.0 can be used.
+   - On some platforms, if the initiator and target nodes have set the `http_proxy` or `https_proxy` environment variables, it will also affect the communication between Transfer Engine and the metadata service.
+
+   Transfer Engine support multiple kinds of metadata services, including `etcd`, `redis`, and `http`. The following describes how to start the metadata service using `etcd` and `http` as examples.
+
+   1.1. **`etcd`**
+
+   For example, the following command line can be used to start the etcd service:
       ```bash
       # This is 10.0.0.1
       etcd --listen-client-urls http://0.0.0.0:2379  --advertise-client-urls http://10.0.0.1:2379
       ```
-   - On some platforms, if the initiator and target nodes have set the `http_proxy` or `https_proxy` environment variables, it will also affect the communication between Transfer Engine and the etcd service, reporting the "Error from etcd client: 14" error.
+
+   1.2. **`http`**
+
+   For example, you can use the `http` service in the `mooncake-transfer-engine/example/http-metadata-server` example:
+      ```bash
+      # This is 10.0.0.1
+      # cd mooncake-transfer-engine/example/http-metadata-server
+      go run . --addr=:8080
+      ```
 
 2. **Start the target node.**
     ```bash
@@ -117,6 +132,7 @@ After successfully compiling Transfer Engine, the test program `transfer_engine_
    - `--mode=target` indicates the start of the target node. The target node does not initiate read/write requests; it passively supplies or writes data as required by the initiator node.
       > Note: In actual applications, there is no need to distinguish between target nodes and initiator nodes; each node can freely initiate read/write requests to other nodes in the cluster.
    - `--metadata_server` is the address of the metadata server (the full address of the etcd service).
+      > Change `--metadata_server` to `--metadata_server=http://10.0.0.1:8080/metadata` and add `--metadata_type=http` when using `http` as the `metadata` service.
    - `--local_server_name` represents the address of this machine, which does not need to be set in most cases. If this option is not set, the value is equivalent to the hostname of this machine (i.e., `hostname(2)`). Other nodes in the cluster will use this address to attempt out-of-band communication with this node to establish RDMA connections.
       > Note: If out-of-band communication fails, the connection cannot be established. Therefore, if necessary, you need to modify the `/etc/hosts` file on all nodes in the cluster to locate the correct node through the hostname.
    - `--device_name` indicates the name of the RDMA network card used in the transfer process.
@@ -414,14 +430,24 @@ Value = {
 ```
 </details>
 
+### HTTP Metadata Server
+
+The HTTP server should implement three following RESTful APIs, while the metadata server configured to `http://host:port/metadata` as an example:
+
+1. `GET /metadata?key=$KEY`: Get the metadata corresponding to `$KEY`.
+2. `PUT /metadata?key=$KEY`: Update the metadata corresponding to `$KEY` to the value of the request body.
+3. `DELETE /metadata?key=$KEY`: Delete the metadata corresponding to `$KEY`.
+
+For specific implementation, refer to the demo service implemented in Golang at [mooncake-transfer-engine/example/http-metadata-server](../../mooncake-transfer-engine/example/http-metadata-server).
+
 ### Initialization
 
 ```cpp
 TransferEngine(std::unique_ptr<TransferMetadata> metadata_client);
 TransferMetadata(const std::string &metadata_server, const std::string &protocol = "etcd");
 ```
 
-- Pointer to a `TransferMetadata` object, which abstracts the communication logic between the TransferEngine framework and the metadata server. We currently support `etcd` and `redis` protocols, while `metadata_server` represents the IP address or hostname of the etcd or redis server.
+- Pointer to a `TransferMetadata` object, which abstracts the communication logic between the TransferEngine framework and the metadata server. We currently support `etcd`, `redis` and `http` protocols, while `metadata_server` represents the IP address or hostname of the etcd or redis server, or the base HTTP URI of http server.
 
 For easy exception handling, TransferEngine needs to call the init function for secondary construction after construction:
 ```cpp

doc/zh/build.md

@@ -31,6 +31,7 @@ $ pip install -e .
 - gcc: 10.2.1
 - go: 1.19+
 - hiredis
+- curl
 
 ### 编译步骤
 
@@ -46,6 +47,7 @@ $ pip install -e .
                    libgtest-dev \
                    libjsoncpp-dev \
                    libnuma-dev \
+                   libcurl4-openssl-dev \
                    libhiredis-dev
 
     # For centos/alibaba linux os
@@ -59,6 +61,8 @@ $ pip install -e .
                 boost-devel \
                 openssl-devel \
                 protobuf-devel \
+                hiredis-devel \
+                libcurl-devel \
                 protobuf-compiler 
     ```
 
@@ -149,4 +153,5 @@ $ pip install -e .
 - `-DUSE_CXL=[ON|OFF]`: 启用 CXL 支持 
 - `-DWITH_P2P_STORE=[ON|OFF]`: 启用 Golang 支持并编译 P2P Store 组件，注意 go 1.19+
 - `-DWITH_WITH_RUST_EXAMPLE=[ON|OFF]`: 启用 Rust 支持
-- `-DWITH_REDIS_=[ON|OFF]`: 启用基于 Redis 的元数据服务
+- `-DUSE_REDIS=[ON|OFF]`: 启用基于 Redis 的元数据服务
+- `-DUSE_HTTP=[ON|OFF]`: 启用基于 Http 的元数据服务

doc/zh/run-examples.md

@@ -62,13 +62,28 @@ Mooncake 支持在执行 `cmake` 命令期间添加下列高级编译选项：
 ## Transfer Engine Bench 使用方法
 编译 Transfer Engine 成功后，可在 `build/mooncake-transfer-engine/example` 目录下产生测试程序 `transfer_engine_bench`。该工具通过调用 Transfer Engine 接口，发起节点从目标节点的 DRAM 处反复读取/写入数据块，以展示 Transfer Engine 的基本用法，并可用于测量读写吞吐率。目前 Transfer Engine Bench 工具可用于 RDMA 协议（GPUDirect 正在测试） 及 TCP 协议。
 
-1. **启动 `etcd` 服务。** 该服务用于 Mooncake 各类元数据的集中高可用管理，包括 Transfer Engine 的内部连接状态等。需确保发起节点和目标节点都能顺利通达该 etcd 服务，因此需要注意：
-   - etcd 服务的监听 IP 不应为 127.0.0.1，需结合网络环境确定。在实验环境中，可使用 0.0.0.0。例如，可使用下列命令行启动合要求的服务：
+1. **启动 `metadata` 服务。** 该服务用于 Mooncake 各类元数据的集中高可用管理，包括 Transfer Engine 的内部连接状态等。需确保发起节点和目标节点都能顺利通达该 metadata 服务，因此需要注意：
+   - metadata 服务的监听 IP 不应为 127.0.0.1，需结合网络环境确定。在实验环境中，可使用 0.0.0.0。
+   - 在某些平台下，如果发起节点和目标节点设置了 `http_proxy` 或 `https_proxy` 环境变量，也会影响 Transfer Engine 与 metadata 服务的通信。
+
+   Transfer Engine 支持多种 metadata 服务，包括 `etcd`, `redis` 和 `http`。下面以 `etcd` 和 `http` 为例说明如何启动 metadata 服务。
+
+   1.1. **`etcd`**
+
+   例如，可使用如下命令行启动 `etcd` 服务：
       ```bash
       # This is 10.0.0.1
       etcd --listen-client-urls http://0.0.0.0:2379 --advertise-client-urls http://10.0.0.1:2379
       ```
-   - 在某些平台下，如果发起节点和目标节点设置了 `http_proxy` 或 `https_proxy` 环境变量，也会影响 Transfer Engine 与 etcd 服务的通信，报告“Error from etcd client: 14”错误。
+
+   1.2. **启动 `http` 作为 `metadata` 服务**
+
+   例如，可使用 `mooncake-transfer-engine/example/http-metadata-server` 示例中的 `http` 服务：
+      ```bash
+      # This is 10.0.0.1
+      # cd mooncake-transfer-engine/example/http-metadata-server
+      go run . --addr=:8080
+      ```
 
 2. **启动目标节点。**
     ```bash
@@ -84,6 +99,7 @@ Mooncake 支持在执行 `cmake` 命令期间添加下列高级编译选项：
    - `--mode=target` 表示启动目标节点。目标节点不发起读写请求，只是被动按发起节点的要求供给或写入数据。
       > 注意：实际应用中可不区分目标节点和发起节点，每个节点可以向集群内其他节点自由发起读写请求。
    - `--metadata_server` 为元数据服务器地址（etcd 服务的完整地址）。
+      > 如果使用 `http` 作为 `metadata` 服务，需要将 `--metadata_server` 参数改为 `--metadata_server=http://10.0.0.1:8080/metadata`，并且指定 `--metadata_type=http`。
    - `--local_server_name` 表示本机器地址，大多数情况下无需设置。如果不设置该选项，则该值等同于本机的主机名（即 `hostname(2)` ）。集群内的其它节点会使用此地址尝试与该节点进行带外通信，从而建立 RDMA 连接。
       > 注意：若带外通信失败则连接无法建立。因此，若有必要需修改集群所有节点的 `/etc/hosts` 文件，使得可以通过主机名定位到正确的节点。
    - `--device_name` 表示传输过程使用的 RDMA 网卡名称。

doc/zh/transfer-engine.md

@@ -68,13 +68,28 @@ Transfer Engine 使用SIEVE算法来管理端点的逐出。如果由于链路
 
 编译 Transfer Engine 成功后，可在 `build/mooncake-transfer-engine/example` 目录下产生测试程序 `transfer_engine_bench`。
 
-1. **启动 `etcd` 服务。** 该服务用于 Mooncake 各类元数据的集中高可用管理，包括 Transfer Engine 的内部连接状态等。需确保发起节点和目标节点都能顺利通达该 etcd 服务，因此需要注意：
-   - etcd 服务的监听 IP 不应为 127.0.0.1，需结合网络环境确定。在实验环境中，可使用 0.0.0.0。例如，可使用下列命令行启动合要求的服务：
+1. **启动 `metadata` 服务。** 该服务用于 Mooncake 各类元数据的集中高可用管理，包括 Transfer Engine 的内部连接状态等。需确保发起节点和目标节点都能顺利通达该 metadata 服务，因此需要注意：
+   - metadata 服务的监听 IP 不应为 127.0.0.1，需结合网络环境确定。在实验环境中，可使用 0.0.0.0。
+   - 在某些平台下，如果发起节点和目标节点设置了 `http_proxy` 或 `https_proxy` 环境变量，也会影响 Transfer Engine 与 metadata 服务的通信。
+
+   Transfer Engine 支持多种 metadata 服务，包括 `etcd`, `redis` 和 `http`。下面以 `etcd` 和 `http` 为例说明如何启动 metadata 服务。
+
+   1.1. **`etcd`**
+
+   例如，可使用如下命令行启动 `etcd` 服务：
       ```bash
       # This is 10.0.0.1
       etcd --listen-client-urls http://0.0.0.0:2379 --advertise-client-urls http://10.0.0.1:2379
       ```
-   - 在某些平台下，如果发起节点和目标节点设置了 `http_proxy` 或 `https_proxy` 环境变量，也会影响 Transfer Engine 与 etcd 服务的通信，报告“Error from etcd client: 14”错误。
+
+   1.2. **启动 `http` 作为 `metadata` 服务**
+
+   例如，可使用 `mooncake-transfer-engine/example/http-metadata-server` 示例中的 `http` 服务：
+      ```bash
+      # This is 10.0.0.1
+      # cd mooncake-transfer-engine/example/http-metadata-server
+      go run . --addr=:8080
+      ```
 
 2. **启动目标节点。**
     ```bash
@@ -91,6 +106,7 @@ Transfer Engine 使用SIEVE算法来管理端点的逐出。如果由于链路
    - `--mode=target` 表示启动目标节点。目标节点不发起读写请求，只是被动按发起节点的要求供给或写入数据。
       > 注意：实际应用中可不区分目标节点和发起节点，每个节点可以向集群内其他节点自由发起读写请求。
    - `--metadata_server` 为元数据服务器地址（etcd 服务的完整地址）。
+      > 如果使用 `http` 作为 `metadata` 服务，需要将 `--metadata_server` 参数改为 `--metadata_server=http://10.0.0.1:8080/metadata`，并且指定 `--metadata_type=http`。
    - `--local_server_name` 表示本机器地址，大多数情况下无需设置。如果不设置该选项，则该值等同于本机的主机名（即 `hostname(2)` ）。集群内的其它节点会使用此地址尝试与该节点进行带外通信，从而建立 RDMA 连接。
       > 注意：若带外通信失败则连接无法建立。因此，若有必要需修改集群所有节点的 `/etc/hosts` 文件，使得可以通过主机名定位到正确的节点。
    - `--device_name` 表示传输过程使用的 RDMA 网卡名称。
@@ -383,6 +399,16 @@ Value = {
 ```
 </details>
 
+### HTTP 元数据服务
+
+使用 HTTP 作为 metadata 元数据服务时，HTTP 服务端需要提供三个接口，以 metadata_server 配置为 `http://host:port/metadata` 举例：
+
+1. `GET /metadata?key=$KEY`：获取 `$KEY` 对应的元数据。
+2. `PUT /metadata?key=$KEY`：更新 `$KEY` 对应的元数据为请求 body 的值。
+3. `DELETE /metadata?key=$KEY`：删除 `$KEY` 对应的元数据。
+
+具体实现，可以参考 [mooncake-transfer-engine/example/http-metadata-server](../../mooncake-transfer-engine/example/http-metadata-server) 用 Golang 实现的 demo 服务。
+
 ### 构造函数与初始化
 
 ```cpp
@@ -391,7 +417,7 @@ TransferMetadata(const std::string &metadata_server, const std::string &protocol
 ```
 
 - TransferMetadata 对象指针，该对象将 TransferEngine 框架与元数据服务器等带外通信逻辑抽取出来，以方便用户将其部署到不同的环境中。
-  目前支持 `etcd` 及 `redis` 两种元数据服务。`metadata_server` 表示元数据服务器/集群的 IP 地址或主机名。
+  目前支持 `etcd`，`redis` 和 `http` 三种元数据服务。`metadata_server` 表示 `etcd`/`redis` 的 IP 地址/主机名，或者 http 服务的 URI。
 
 为了便于异常处理，TransferEngine 在完成构造后需要调用init函数进行二次构造：
 ```cpp

mooncake-transfer-engine/example/http-metadata-server/go.mod

@@ -0,0 +1,34 @@
+module github.com/kvcache-ai/Mooncake/mooncake-transfer-engine/example/http-metadata-server
+
+go 1.22.9
+
+require github.com/gin-gonic/gin v1.10.0
+
+require (
+	github.com/bytedance/sonic v1.11.6 // indirect
+	github.com/bytedance/sonic/loader v0.1.1 // indirect
+	github.com/cloudwego/base64x v0.1.4 // indirect
+	github.com/cloudwego/iasm v0.2.0 // indirect
+	github.com/gabriel-vasile/mimetype v1.4.3 // indirect
+	github.com/gin-contrib/sse v0.1.0 // indirect
+	github.com/go-playground/locales v0.14.1 // indirect
+	github.com/go-playground/universal-translator v0.18.1 // indirect
+	github.com/go-playground/validator/v10 v10.20.0 // indirect
+	github.com/goccy/go-json v0.10.2 // indirect
+	github.com/json-iterator/go v1.1.12 // indirect
+	github.com/klauspost/cpuid/v2 v2.2.7 // indirect
+	github.com/leodido/go-urn v1.4.0 // indirect
+	github.com/mattn/go-isatty v0.0.20 // indirect
+	github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
+	github.com/modern-go/reflect2 v1.0.2 // indirect
+	github.com/pelletier/go-toml/v2 v2.2.2 // indirect
+	github.com/twitchyliquid64/golang-asm v0.15.1 // indirect
+	github.com/ugorji/go/codec v1.2.12 // indirect
+	golang.org/x/arch v0.8.0 // indirect
+	golang.org/x/crypto v0.23.0 // indirect
+	golang.org/x/net v0.25.0 // indirect
+	golang.org/x/sys v0.20.0 // indirect
+	golang.org/x/text v0.15.0 // indirect
+	google.golang.org/protobuf v1.34.1 // indirect
+	gopkg.in/yaml.v3 v3.0.1 // indirect
+)