Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: add initial support for an sqlite oplog store #499

Merged
merged 3 commits into from
Sep 26, 2024

Conversation

garethgeorge
Copy link
Owner

@garethgeorge garethgeorge commented Sep 26, 2024

This adds initial support for an sqlite oplog store. Future work will be needed to add migrations and move over to sqlite for a more consistent data store.

Bbolt so far has been very reliable, but is not particularly standard whereas sqlite offers an easily understandable schema for contributors. As backrest matures maintainability becomes a priority.

Some datapoints --

goos: linux
goarch: amd64
pkg: github.com/garethgeorge/backrest/internal/oplog/storetests
cpu: AMD Ryzen 9 5900X 12-Core Processor            
BenchmarkAdd/bbolt-24               7914           1467258 ns/op
BenchmarkAdd/memory-24          25463836               630.0 ns/op
BenchmarkAdd/sqlite-24             14718            815904 ns/op
BenchmarkList/100/sqlite-24                72892            171401 ns/op
BenchmarkList/100/bbolt-24                120385             99851 ns/op
BenchmarkList/100/memory-24               180612             66349 ns/op
BenchmarkList/1000/bbolt-24                10000           1013055 ns/op
BenchmarkList/1000/memory-24               16929            728347 ns/op
BenchmarkList/1000/sqlite-24                8418           1425632 ns/op
BenchmarkList/10000/bbolt-24                 909          13382735 ns/op
BenchmarkList/10000/memory-24               1551           7337943 ns/op
BenchmarkList/10000/sqlite-24                866          14544289 ns/op
BenchmarkGetLastItem/100/bbolt-24        1677454              7218 ns/op
BenchmarkGetLastItem/100/memory-24       1643539              7583 ns/op
BenchmarkGetLastItem/100/sqlite-24        479854             25782 ns/op
BenchmarkGetLastItem/1000/bbolt-24        198956             58122 ns/op
BenchmarkGetLastItem/1000/memory-24       123151             99884 ns/op
BenchmarkGetLastItem/1000/sqlite-24       438768             26287 ns/op
BenchmarkGetLastItem/10000/bbolt-24        21298            555980 ns/op
BenchmarkGetLastItem/10000/memory-24               10000           1089095 ns/op
BenchmarkGetLastItem/10000/sqlite-24              438074             25319 ns/op
PASS
ok      github.com/garethgeorge/backrest/internal/oplog/storetests      355.271s

Benchmark results suggest that sqlite is at very much "fast enough". Operations are slightly slower in some cases with small numbers of operations in the log, but scale much better overall. The "get last item" results are of particular interest as this is the operation that's polled frequently when scheduling tasks.

@garethgeorge garethgeorge merged commit 50b4be7 into main Sep 26, 2024
7 checks passed
@garethgeorge garethgeorge deleted the garethgeorge/sqlite-store branch September 27, 2024 20:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant