diff --git a/.markdownlint.yml b/.markdownlint.yml index 6eecf8a..34be2be 100644 --- a/.markdownlint.yml +++ b/.markdownlint.yml @@ -1,5 +1,6 @@ MD010: false MD013: line_length: 1024 +MD024: false MD046: style: "fenced" diff --git a/releases/2.4.0/README-zh.md b/releases/2.4.0/README-zh.md new file mode 100644 index 0000000..dbeabd8 --- /dev/null +++ b/releases/2.4.0/README-zh.md @@ -0,0 +1,124 @@ +# Dragonfly v2.4.0 发布了 + +Dragonfly v2.4.0 发布了!🎉🎉🎉 感谢为此版本做出贡献的[贡献者们](https://github.com/dragonflyoss/dragonfly/graphs/contributors),欢迎访问 [d7y.io](https://d7y.io) 网站。 + +![dragonfly](images/dragonfly-linear.png) + +## 新增特性 + +### 负载感知调度算法 + +调度算法升级为一种两阶段调度算法,结合中央调度与节点级二次调度,基于实时负载感知来优化 P2P 下载性能。 + +![p1](images/p1.svg) + +更多信息,请参阅 [Scheduling](https://d7y.io/docs/next/operations/deployment/applications/scheduler/#scheduling)。 + +### Vortex 协议支持 P2P 文件传输 + +Dragonfly 提供了基于 TLV 的新型 Vortex 传输协议,以提升 P2P 下载性能。使用 TLV(Tag-Length-Value)格式作为轻量级协议,替代 gRPC 进行节点间的数据传输。 +与 gRPC 相比,基于 TCP 的 Vortex 可将大文件下载时间减少 50%,基于 QUIC 的 Vortex 可减少 40%,两者都能有效降低峰值内存使用。 + +更多信息,请参阅 [TCP Protocol Support for P2P File Transfer](https://github.com/dragonflyoss/design/blob/main/systems-analysis/tcp-protocol/tcp-protocol.md) 和 [QUIC Protocol Support for P2P File Transfer](https://github.com/dragonflyoss/design/blob/main/systems-analysis/quic-protocol/quic-protocol.md)。 + +### Request SDK + +新增 SDK 用于将用户请求路由到 Seed Peer,使用一致性哈希算法,取代之前的 Kubernetes Service 负载均衡方式。 + +![p2](images/p2.svg) + +更多信息,请参阅 [Request SDK](https://d7y.io/docs/next/advanced-guides/request-sdk/)。 + +### 指定集群 ID 实现多集群 Kubernetes 简化部署 + +Dragonfly 支持一种简化功能,用于部署和管理多个 Kubernetes 集群,方法是为每个集群显式分配一个 `schedulerClusterID`。 +这种方式允许用户直接控制集群亲和性,而无需依赖基于位置的调度元数据,例如 IDC、主机名或 IP 地址。 + +使用此功能,每个 Peer、Seed Peer 和 Scheduler 都通过明确定义的 Scheduler Cluster ID 来确定其目标调度器集群。 +这确保了集群之间的精确隔离以及可预测的跨集群行为。 + +![p3](images/p3.png) + +更多信息,请参阅 [Create Dragonfly Cluster Simple](https://d7y.io/docs/next/getting-started/quick-start/multi-cluster-kubernetes/#create-dragonfly-cluster-simple)。 + +### Manager 和 Scheduler 组件的性能和资源优化 + +增强了 Manager 和 Scheduler 组件的性能和资源利用率,同时显著降低了 CPU 和内存开销,提升了系统效率和资源管理能力。 + +### 增强预热功能 + +- 支持预热到指定节点:采用基于优先级的选择逻辑,其中 IP 指定具有最高优先级,其次是基于数量的选择,最后是基于百分比的选择。 + +- 支持在单个预热任务中预热多个 URL。 + +- 支持通过 Scheduler gRPC 接口预热文件和镜像。 + +![p4](images/p4.png) + +### 基于镜像 Blob SHA256 计算 ID 以避免重复下载 + +客户端现已支持直接根据镜像 Blob 的 SHA256 哈希值计算任务 ID,而不再使用下载 URL。此增强功能可以防止当同一 Blob 从不同镜像仓库 +域名访问时产生的重复下载和数据冗余问题,新版本默认启用此功能。 + +### 缓存 HTTP 307 重定向 URL + +新增对 HTTP 307(临时重定向)响应 URL 的缓存支持,以优化 Dragonfly 的多分片下载性能。当下载 URL 被拆分为多个分片时, +重定向目标地址现在会被缓存,从而消除冗余的重定向请求并降低延迟。 + +### Go 客户端已弃用,并由 Rust 客户端取代 + +Go 客户端已被弃用,并由 [Rust 客户端](https://github.com/dragonflyoss/client) 取代。未来所有的开发和维护工作将专注于 Rust 客户端,它提供了更好的性能、稳定性和可靠性。 + +更多信息,请参阅 [dragoflyoss/client](https://github.com/dragonflyoss/client)。 + +### 附加功能增强 + +- 在 Rust 客户端中启用 ARM64 架构的 64K 页面大小支持。 +- 修复 dfget 版本输出中缺失的 git commit 元数据问题。 +- 支持 containerd V3 配置中 `io.containerd.cri.v1.images` 插件的 `config_path` 配置。 +- 在 `reqwest` 中使用 `hickory-dns` 替换 `glibc DNS` 解析器,以实现 DNS 缓存并防止分片下载过程中过多的 DNS 查询。 +- 支持 `--include-files` 配置参数,可选择性地从目录中下载文件。 +- 添加 `--no-progress` 配置参数,可禁用下载进度条输出。 +- 支持 `backend` 操作中的自定义请求头,为 HTTP 请求提供灵活的请求头配置。 +- 重构日志输出,减少冗余日志记录并提高日志可读性。 + +## 重要修复 + +- 将数据库字段类型从 `text` 修改为 `longtext`,以支持存储预热任务的信息。 +- 修复了 Scheduler 关闭期间重复停止 Seed Peer 服务时导致的 panic 问题。 +- 修复了在指定 Redis 密码但未设置用户名时,Broker 认证失败的问题。 + +## Nydus + +### 新增特性 + +- Nydusd:为 RAFS V5 和 V6 格式添加运行时 CRC32 校验支持,增强数据完整性验证。 +- Nydusd:支持在 Nydusd 恢复过程中重新发送 FUSE 请求,提高进程恢复的可靠性。 +- Nydusd:增强 VFS 状态保存机制,用于进程热升级和故障转移。 +- Nydusify:支持 Nydus 到 OCI 的反向转换能力(实验性功能),支持无缝迁移回 OCI 格式。 +- Nydusify:实现镜像拷贝时零磁盘传输,显著减少复制操作期间对本地磁盘的使用。 +- Snapshotter:将 `blob.meta` 打包到 Bootstrap 中,提高 RAFS V6 镜像的 Blob 下载时的可靠性 + +### 重要修复 + +- Nydusd:修复镜像仓库认证中 `access_token` 字段的认证令牌获取问题。 +- Nydusd:为 umount API 添加了递归的 inode/dentry 失效处理。 +- Nydus Image:修复 optimize 子命令中的多个问题,并添加后端配置支持。 +- Snapshotter:为代理模式实现延迟父快照恢复,修复父快照缺失问题。 + +## 其他 + +您可以在 [CHANGELOG](https://github.com/dragonflyoss/dragonfly/blob/main/CHANGELOG.md) 中查看更多详细信息。 + +## 链接 + +- Dragonfly 网站: +- Dragonfly 仓库: +- Dragonfly 客户端仓库: +- Dragonfly 控制台仓库: +- Dragonfly Charts 仓库: +- Dragonfly 监控仓库: + +## Dragonfly Github + +![qrcode](images/qrcode.webp) diff --git a/releases/2.4.0/README.md b/releases/2.4.0/README.md new file mode 100644 index 0000000..ffb6655 --- /dev/null +++ b/releases/2.4.0/README.md @@ -0,0 +1,131 @@ +# Dragonfly v2.4.0 is released + +Dragonfly v2.4.0 is released!🎉🎉🎉 Thanks the [contributors](https://github.com/dragonflyoss/dragonfly/graphs/contributors) who made this release happend and welcome you to visit [d7y.io](https://d7y.io) website. + +![dragonfly](images/dragonfly-linear.png) + +## New features and enhancements + +### load-aware scheduling algorithm + +A two-stage scheduling algorithm combining central scheduling with node-level secondary scheduling to +optimize P2P download performance based on real-time load awareness. + +![p1](images/p1.svg) + +For more information, please refer to the [Scheduling](https://d7y.io/docs/next/operations/deployment/applications/scheduler/#scheduling). + +### Vortex Protocol Support for P2P File Transfer + +Dragonfly provide the new Vortex transfer protocol based on TLV to improve the download performance in internal network. +Use the TLV (Tag-Length-Value) format as a lightweight protocol to replace gRPC for data transfer between peers. +TCP-based Vortex reduces large file download time by `50%` and QUIC-based Vortex by `40%` compared to gRPC, both effectively reducing peak memory usage. + +For more information, please refer to the [TCP Protocol Support for P2P File Transfer](https://github.com/dragonflyoss/design/blob/main/systems-analysis/tcp-protocol/tcp-protocol.md) and [QUIC Protocol Support for P2P File Transfer](https://github.com/dragonflyoss/design/blob/main/systems-analysis/quic-protocol/quic-protocol.md). + +### Request SDK + +A SDK for routing User requests to Seed Peers using consistent hashing, replacing the previous Kubernetes Service load balancing approach. + +![p2](images/p2.svg) + +For more details, please refer to [Request SDK](https://d7y.io/docs/next/advanced-guides/request-sdk/). + +### Simple Multi‑Cluster Kubernetes Deployment with Scheduler Cluster ID + +Dragonfly supports a simplified feature for deploying and managing multiple Kubernetes clusters by explicitly assigning a `schedulerClusterID` to each cluster. +This approach allows users to directly control cluster affinity without relying on location‑based scheduling metadata such as IDC, hostname, or IP. + +Using this feature, each Peer, Seed Peer, and Scheduler determines its target scheduler cluster through a clearly defined scheduler cluster ID. +This ensures precise separation between clusters and predictable cross‑cluster behavior. + +![p3](images/p3.png) + +For more information, please refer to the [Create Dragonfly Cluster Simple](https://d7y.io/docs/next/getting-started/quick-start/multi-cluster-kubernetes/#create-dragonfly-cluster-simple). + +### Performance and Resource Optimization for Manager and Scheduler Components + +Enhanced service performance and resource utilization across Manager and Scheduler components while significantly reducing +CPU and memory overhead, delivering improved system efficiency and better resource management. + +### Enhanced Preheating + +- Support for IP-based peer selection in preheating jobs with priority-based selection logic where IP specification + takes highest priority, followed by count-based and percentage-based selection + +- Support for preheating multiple URLs in a single request. + +- Support for preheating file and image via Scheduler gRPC interface. + +![p4](images/p4.png) + +### Calculate task ID based on image blob SHA256 to avoid redundant downloads + +The Client now supports calculating task IDs directly from the SHA256 hash of image blobs instead of using the download URL. +This enhancement prevents redundant downloads and data duplication when the same blob is accessed from +different registry domains. + +### Cache HTTP 307 redirects for split downloads + +Support for caching HTTP 307 (Temporary Redirect) responses to optimize Dragonfly's multi-piece download performance. +When a download URL is split into multiple pieces, the redirect target is now cached, +eliminating redundant redirect requests and reducing latency. + +### Go Client Deprecated and Replaced by Rust Client + +The Go client has been deprecated and replaced by the [Rust Client](https://github.com/dragonflyoss/client). All future development and maintenance will focus +exclusively on the Rust client, which offers improved performance, stability, and reliability. + +For more information, please refer to the [dragoflyoss/client](https://github.com/dragonflyoss/client). + +### Additional Enhancements + +- Enable 64K page size support for ARM64 in the Dragonfly Rust client. +- Fix missing git commit metadata in dfget version output. +- Support for `config_path` of `io.containerd.cri.v1.images` plugin for containerd V3 configuration. +- Replaces glibc DNS resolver with hickory-dns in reqwest to implement DNS caching and prevent excessive DNS lookups during piece downloads. +- Support for the --include-files flag to selectively download files from a directory. +- Add the `--no-progress flag` to disable the download progress bar output. +- Support for custom request headers in backend operations, enabling flexible header configuration for HTTP requests. +- Refactored log output to reduce redundant logging and improve overall logging efficiency. + +## Significant bug fixes + +- Modified the database field type from `text` to `longtext` to support storing the information of preheating job. +- Fixed panic on repeated seed peer service stops during Scheduler shutdown. +- Fixed broker authentication failure when specifying the Redis password without setting a username. + +## Nydus + +### New features and enhancements + +- Nydusd: Add CRC32 validation support for both RAFS V5 and V6 formats, enhancing data integrity verification. +- Nydusd: Support resending FUSE requests during nydusd restoration, improving daemon recovery reliability. +- Nydusd: Enhance VFS state saving mechanism for daemon hot upgrade and failover. +- Nydusify: Introduce Nydus-to-OCI reverse conversion capability, enabling seamless migration back to OCI format. +- Nydusify: Implement zero-disk transfer for image copy, significantly reducing local disk usage during copy operations. +- Snapshotter: Builtin blob.meta in bootstrap for blob fetch reliability for RAFS v6 image. + +### Significant bug fixes + +- Nydusd: Fix auth token fetching for `access_token` field in registry authentication. +- Nydusd: Add recursive inode/dentry invalidation for umount API. +- Nydus Image: Fix multiple issues in optimize subcommand and add backend configuration support. +- Snapshotter: Implement lazy parent recovery for proxy mode to handle missing parent snapshots. + +## Others + +You can see [CHANGELOG](https://github.com/dragonflyoss/dragonfly/blob/main/CHANGELOG.md) for more details. + +## Links + +- Dragonfly Website: +- Dragonfly Repository: +- Dragonfly Client Repository: +- Dragonfly Console Repository: +- Dragonfly Charts Repository: +- Dragonfly Monitor Repository: + +## Dragonfly Github + +![qrcode](images/qrcode.webp) diff --git a/releases/2.4.0/images/dragonfly-linear.png b/releases/2.4.0/images/dragonfly-linear.png new file mode 100644 index 0000000..3341b08 Binary files /dev/null and b/releases/2.4.0/images/dragonfly-linear.png differ diff --git a/releases/2.4.0/images/p1.svg b/releases/2.4.0/images/p1.svg new file mode 100644 index 0000000..72163a8 --- /dev/null +++ b/releases/2.4.0/images/p1.svg @@ -0,0 +1,4 @@ + + + +
P2P Scheduling FLow
P2P Scheduling FLow
Peer
Peer
Parent A
40%
Parent A...
Parent B
35%
Parent B...
Parent N
n%
Parent N...
Stage 1
Stage 1
Stage 2
Stage 2
Secondary Scheduling (Per-Piece Selection)
Secondary Scheduling (Per-Piece Selection)
Scheduler
Scheduler

Central Scheduling (Top N Parents)

Central Scheduling (Top N Parents)
\ No newline at end of file diff --git a/releases/2.4.0/images/p2.svg b/releases/2.4.0/images/p2.svg new file mode 100644 index 0000000..bcc1a45 --- /dev/null +++ b/releases/2.4.0/images/p2.svg @@ -0,0 +1,4 @@ + + + +
Request SDK
Seed Peer 1
Seed Peer 2
Seed Peer 3

OCI registry
Layer1
Request
Chunk 1
Chunk 2
Chunk 3
Layer1
User
\ No newline at end of file diff --git a/releases/2.4.0/images/p3.png b/releases/2.4.0/images/p3.png new file mode 100644 index 0000000..4ccf5b5 Binary files /dev/null and b/releases/2.4.0/images/p3.png differ diff --git a/releases/2.4.0/images/p4.png b/releases/2.4.0/images/p4.png new file mode 100644 index 0000000..a41aa9e Binary files /dev/null and b/releases/2.4.0/images/p4.png differ diff --git a/releases/2.4.0/images/qrcode.webp b/releases/2.4.0/images/qrcode.webp new file mode 100644 index 0000000..f66d14b Binary files /dev/null and b/releases/2.4.0/images/qrcode.webp differ