ROADMAP.md

Roadmap

• Q1: Preliminary DuckDB extension (with limited subset of planned features)
• Q2: PuffinDB 0.1 with basic distributed query planner (filter pushdown)
• Q4: PuffinDB 1.0 with advanced distributed query planner (including support for full TPC-H benchmark)
• 2024: PuffinDB 2.0 (including support for full TPC-DS benchmark)

Features

Features will be implemented in the following order. Please start an Idea discussion to discuss any element of this proposed roadmap.

Project Framework

• Domain name
• Logo design
• Core architecture design
• Temporary GitHub repository
• Permanent GitHub organization
• DuckDB extension repository (private for now)
• PuffinDB engine repository (private for now)
• Continuous integration framework
• Unit testing framework
• Integration testing framework

DuckDB Extension

• Project website
• DuckDB extension (with minimal set of features)
• Airbyte connector framework
• Support for SELECT THROUGH syntax
• Authentication
• Authorization

Iceberg Integration

• Catalog serverless function
• Lakehouse template
• Lakehouse catalog integration (AWS Glue Data Catalog, Amazon DynamoDB, and Amazon RDS)
• Read queries against Lakehouse tables executed by DuckDB
• Write queries against Lakehouse tables executed by Amazon Athena

Basic Distributed Query Engine

• Engine serverless function
• Engine template
• Remote query engine running on AWS Lambda functions
• Query logs recorded as JSON values in Redis cluster (using Amazon ElastiCache for Redis)

Query Proxy

• Query proxy
• PRQL to SQL translator
• Malloy to SQL translator
• SQL dialect converter

Advanced Distributed Query Engine

• Remote query engine running on Monostore
• Basic distributed query planner
• Partition caching on AWS Lambda function
• Query result caching on Object Store
• Query result caching on CDN (Amazon CloudFront)

Other Features

• AWS Marketplace provisioning
• Microsoft Azure support
• Joins across tables managed by different Lakehouse instances
• Asynchronous invocations over Apache Arrow
• Arrow Database Connectivity support
• Concurrent suport for multiple Lakehouse instances
• Advanced distributed query planner
• Delta Lake support
• Apache Hudi support
• Joins across heterogenous tables using different table formats
• AWS Fargate support
• Google Cloud support