Skip to content

Commit

Permalink
Add loom simulations of spfs clean data race
Browse files Browse the repository at this point in the history
    $ cargo test --bin clean-loom

This shows three variations on how `spfs clean` can race with writers to
a repo.

The first one tries to simulate the existing behavior and the
test fails as expected.

The other two demonstrate how a write-ahead log might prevent races. The
first one requires the write-ahead log to be locked while reading in all
the tags, which in turn would block writes for a significant amount of
time.

The second one proposes how the write-ahead log could be pruned over
time with a kakfa-like tombstone concept, and only requires the log to
be locked while actually deleting files.

In either case, the time while the write-ahead log is locked could early
exit in order to not block writers for an extended amount of time. In
this way it could make incremental progress by deleting some number of
objects in each pass.

Signed-off-by: J Robert Ray <jrray@jrray.org>
  • Loading branch information
jrray committed Jul 2, 2024
1 parent dc5428a commit 28130af
Show file tree
Hide file tree
Showing 4 changed files with 308 additions and 32 deletions.
138 changes: 107 additions & 31 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ members = [
"crates/spk-solve",
"crates/spk-solve/crates/*",
"crates/spk-storage",
"crates/spk",
"crates/spk", "clean-loom",
]
resolver = "2"

Expand Down
11 changes: 11 additions & 0 deletions clean-loom/Cargo.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
[package]
name = "clean-loom"
authors.workspace = true
edition.workspace = true
version.workspace = true

[dependencies]
loom = "0.7"

[lints]
workspace = true
Loading

0 comments on commit 28130af

Please sign in to comment.