Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ByteHTAP: ByteDance’s HTAP System with High Data Freshness and Strong Data Consistency #23

Open
mrdrivingduck opened this issue May 18, 2023 · 2 comments
Open

ByteHTAP: ByteDance’s HTAP System with High Data Freshness and Strong Data Consistency #23

mrdrivingduck opened this issue May 18, 2023 · 2 comments
Assignees
@mrdrivingduck
Labels
area/database-management-system DBMS topic/htap Hybrid Transactional and Analytical Processing

Comments

@mrdrivingduck
Copy link
Owner

p3411-chen.pdf

字节跳动根据其商业场景而设计的 HTAP 产品：

  • 支持 PB 级别数据规模
  • 实时性接近分别使用 OLTP / OLAP
  • 秒级数据新鲜度
  • 数据强一致
@mrdrivingduck mrdrivingduck added area/database-management-system DBMS topic/htap Hybrid Transactional and Analytical Processing labels May 18, 2023
@mrdrivingduck mrdrivingduck self-assigned this May 18, 2023
@mrdrivingduck
Copy link
Owner Author

Architecture

OLTP 引擎

使用 ByteNDB，是一个计算与存储分离，一写多读共享存储的架构，类似 Aurora。

image

存储上分为 Page Store 和 Log Store，合并起来就是正确版本的数据。

HTAP 引擎

image

使用引擎分离、共享存储的架构设计。通过统一的 API，经过一个智能代理，把请求分流到 OLTP 或 OLAP 引擎上。这样可以避免 OLTP 和 OLAP 的互相干扰。OLTP 的 DML 操作产生的逻辑日志会被不断地发送到列式存储上。列式存储使用内存增量存储 + 持久化基本存储。

不支持事务内 DML + OLAP 只读查询。

@mrdrivingduck
Copy link
Owner Author

一些产品功能

  1. 允许已有的 ByteNDB 无缝升级
  2. 支持多个 OLTP 数据库同步到同一个 OLAP 引擎，实现跨库分析
  3. 高效的数据迁移
  4. OLAP 引擎加强

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mrdrivingduck mrdrivingduck

Labels
area/database-management-system DBMS topic/htap Hybrid Transactional and Analytical Processing
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@mrdrivingduck