Skip to content

Introduce a WAL, and write data in memory which gets saved to delta in 10mins interval, to reduce the small files problem. #12

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
tonyalaribe merged 40 commits into from
Dec 29, 2025
Merged

Introduce a WAL, and write data in memory which gets saved to delta in 10mins interval, to reduce the small files problem. #12

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
40 commits
Select commit Hold shift + click to select a range
cd78a3c
Add InfluxDB-inspired in-memory buffer with WAL
tonyalaribe Dec 27, 2025
f71406e
Fix clippy warnings
tonyalaribe Dec 27, 2025
cbe3ae6
Remove unused tempfile import in wal.rs
tonyalaribe Dec 27, 2025
c6a4498
Fix WAL and buffered write layer issues
tonyalaribe Dec 27, 2025
bfa8506
Add in-memory UPDATE/DELETE support for buffered write layer
tonyalaribe Dec 27, 2025
f823e26
Refactor DmlExec to use builder pattern to fix clippy warnings
tonyalaribe Dec 27, 2025
cf86ea7
Improve DML routing: skip Delta for uncommitted data
tonyalaribe Dec 27, 2025
c43a239
Fix WAL recovery, schema validation, timestamp bucketing, and shutdow…
tonyalaribe Dec 27, 2025
47e3a85
Fix critical issues: checkpoint ordering, memory limits, error handling
tonyalaribe Dec 27, 2025
2a80e9e
Fix formatting and clippy warning for CI
tonyalaribe Dec 27, 2025
8e43cd8
Fix infinite loop in WAL read on I/O error
tonyalaribe Dec 27, 2025
7d2c403
Trigger CI
tonyalaribe Dec 27, 2025
e017e83
Add WALRUS_DATA_DIR to CI and local test config
tonyalaribe Dec 27, 2025
e919757
Disable Foyer cache in CI to fix test hangs
tonyalaribe Dec 27, 2025
9b44f80
Fix infinite loop in WAL recovery by using checkpoint=true
tonyalaribe Dec 28, 2025
8e914d3
Enable Foyer cache in CI with small disk sizes
tonyalaribe Dec 28, 2025
52006ef
Centralize env var handling with envy crate
tonyalaribe Dec 28, 2025
f090f9d
Fix bounds check, add WAL corruption threshold, cleanup
tonyalaribe Dec 28, 2025
990efdd
Change default buffer retention from 90 to 70 minutes
tonyalaribe Dec 28, 2025
5181509
Fix clippy warnings: remove needless borrows, collapse nested ifs
tonyalaribe Dec 28, 2025
1dad8c2
fmt
tonyalaribe Dec 28, 2025
ac76b13
Add retry limit to memory reservation, improve type checks
tonyalaribe Dec 28, 2025
173751e
Document magic numbers and unsafe env var usage in tests
tonyalaribe Dec 28, 2025
e7b222a
Document RecordBatch clone is cheap (Arc-based, O(columns))
tonyalaribe Dec 28, 2025
1e5df5e
Optimize schema validation and parallelize bucket flushing
tonyalaribe Dec 28, 2025
d49d2e9
Fix flush ordering and document negative timestamp behavior
tonyalaribe Dec 28, 2025
8a94908
Document Delta callback contract: must complete commit before returning
tonyalaribe Dec 28, 2025
1ed53be
Clarify RecordBatch clone overhead: ~100 bytes/batch, not data size
tonyalaribe Dec 28, 2025
010e1b2
Add WAL support for DELETE and UPDATE operations
tonyalaribe Dec 28, 2025
cb830b0
Fix test isolation and config parsing issues
tonyalaribe Dec 29, 2025
4d66cb5
Refactor config to use explicit passing instead of OnceLock
tonyalaribe Dec 29, 2025
e50add9
Refactor for code conciseness: -471 lines
tonyalaribe Dec 29, 2025
5de70b4
Reduce code duplication with helper methods and macros
tonyalaribe Dec 29, 2025
ab5fdac
Replace perform_dml_with_buffer with DmlContext struct
tonyalaribe Dec 29, 2025
9f66e44
Fix statistics test: add missing page_row_limit argument
tonyalaribe Dec 29, 2025
e890a5e
fix fmt
tonyalaribe Dec 29, 2025
b9e6d84
Fix DML tests failing due to global config caching
tonyalaribe Dec 29, 2025
d86e6ae
update documentation and flatten the dashmap usage
tonyalaribe Dec 29, 2025
3a1baab
Remove unnecessary RwLock from TableBuffer.schema
tonyalaribe Dec 29, 2025
a176e0a
fmt
tonyalaribe Dec 29, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .env.minio
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,9 @@ MAX_PG_CONNECTIONS=100
# MinIO doesn't need DynamoDB locking, use local locking
AWS_S3_LOCKING_PROVIDER=""

# WAL storage directory for walrus-rust
WALRUS_DATA_DIR=/tmp/walrus-wal

# Foyer cache configuration for tests
TIMEFUSION_FOYER_MEMORY_MB=256
TIMEFUSION_FOYER_DISK_GB=10
Expand Down
11 changes: 7 additions & 4 deletions .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -62,10 +62,13 @@ jobs:
ENABLE_BATCH_QUEUE: "true"
MAX_PG_CONNECTIONS: "100"
AWS_S3_LOCKING_PROVIDER: ""
TIMEFUSION_FOYER_MEMORY_MB: "256"
TIMEFUSION_FOYER_DISK_GB: "10"
TIMEFUSION_FOYER_TTL_SECONDS: "300"
TIMEFUSION_FOYER_SHARDS: "8"
WALRUS_DATA_DIR: /tmp/walrus-wal
# Use small cache sizes for CI tests (similar to test_config in object_store_cache.rs)
TIMEFUSION_FOYER_MEMORY_MB: "10"
TIMEFUSION_FOYER_DISK_MB: "50"
TIMEFUSION_FOYER_METADATA_MEMORY_MB: "10"
TIMEFUSION_FOYER_METADATA_DISK_MB: "50"
TIMEFUSION_FOYER_SHARDS: "2"
services:
minio:
image: public.ecr.aws/bitnami/minio:latest
Expand Down
1 change: 1 addition & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,3 +9,4 @@ minio
dis-newstyle
*.log
.DS_Store
wal_files/
37 changes: 37 additions & 0 deletions Cargo.lock
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

8 changes: 7 additions & 1 deletion Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ edition = "2024"
[dependencies]
tokio = { version = "1.48", features = ["full"] }
datafusion = "51.0.0"
datafusion-datasource = "51.0.0"
arrow = "57.1.0"
arrow-json = "57.1.0"
uuid = { version = "1.17", features = ["v4", "serde"] }
Expand Down Expand Up @@ -68,8 +69,12 @@ ahash = "0.8"
lru = "0.16.1"
serde_bytes = "0.11.19"
dashmap = "6.1"
envy = "0.4"
tdigests = "1.0"
bincode = "2.0"
bincode = { version = "2.0", features = ["serde"] }
walrus-rust = "0.2.0"
thiserror = "2.0"
strum = { version = "0.27", features = ["derive"] }

[dev-dependencies]
sqllogictest = { git = "https://github.com/risinglightdb/sqllogictest-rs.git" }
Expand All @@ -78,6 +83,7 @@ datafusion-common = "51.0.0"
tokio-postgres = { version = "0.7.10", features = ["with-chrono-0_4"] }
scopeguard = "1.2.0"
rand = "0.9.2"
tempfile = "3"

[features]
default = []
Expand Down
Loading
Loading