-
Notifications
You must be signed in to change notification settings - Fork 3
Fix database file size not shrinking when document_ttl
triggers squash
#1
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix database file size not shrinking when document_ttl
triggers squash
#1
Conversation
Should we also add logic to turn on full auto-vacuum and run an initial VACUUM for existing .db files? |
I don't know if it's needed, actually a test would be helpful. |
Had to loop ~60 times in test since small updates don’t grow SQLite file size until a full page fills. |
Locally it passes with 60 loops; in CI, it passes on macOS but is failing on Ubuntu — working on a more robust solution. |
tests/test_store.py
Outdated
Path(self.db_path).unlink(missing_ok=True) | ||
except PermissionError: | ||
# If db is still open, wait a bit and try again | ||
sleep(0.1) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
sleep
is async, so this has no effect.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
My bad — we also don't need this handling because I was getting a permission error due to not properly closing the database in my test. It wasn't because the database was taking time to close.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks.
It looks like it picked up a conflict due to refactor of files |
…ash (y-crdt#1) * Run VACUUM after document_ttl squash to reclaim freed pages * test-for-size-reduction * make-test-robust * fix-typo * improve-tests * small-refactor * run-pre-commit * remove-sleep-and-delete
* fixtemp-db-connection * add-test * use-tg-in-test * fix-test * fix-test * add-compression-to-yupdates * make-backward-compatible * add-compression-decompression-callbacks * remove-none * remove-staticmethod * ignore-type * fix-conflict-manually * Update pyproject.toml Co-authored-by: David Brochart <david.brochart@gmail.com> * set--as-default * resolve_conflict * refactor_code * modify-test * Update pyproject.toml Co-authored-by: David Brochart <david.brochart@gmail.com> * modify-test * fix * Fix database file size not shrinking when `document_ttl` triggers squash (#1) * Run VACUUM after document_ttl squash to reclaim freed pages * test-for-size-reduction * make-test-robust * fix-typo * improve-tests * small-refactor * run-pre-commit * remove-sleep-and-delete * Drop Jupyter Releaser (#14) * Drop Jupyter Releaser * Remove check_release * Release v0.1.1 * retain-test * fix-test * fix-lint --------- Co-authored-by: David Brochart <david.brochart@gmail.com>
…ash (y-crdt#1) * Run VACUUM after document_ttl squash to reclaim freed pages * test-for-size-reduction * make-test-robust * fix-typo * improve-tests * small-refactor * run-pre-commit * remove-sleep-and-delete
* fixtemp-db-connection * add-test * use-tg-in-test * fix-test * fix-test * add-compression-to-yupdates * make-backward-compatible * add-compression-decompression-callbacks * remove-none * remove-staticmethod * ignore-type * fix-conflict-manually * Update pyproject.toml Co-authored-by: David Brochart <david.brochart@gmail.com> * set--as-default * resolve_conflict * refactor_code * modify-test * Update pyproject.toml Co-authored-by: David Brochart <david.brochart@gmail.com> * modify-test * fix * Fix database file size not shrinking when `document_ttl` triggers squash (y-crdt#1) * Run VACUUM after document_ttl squash to reclaim freed pages * test-for-size-reduction * make-test-robust * fix-typo * improve-tests * small-refactor * run-pre-commit * remove-sleep-and-delete * Drop Jupyter Releaser (y-crdt#14) * Drop Jupyter Releaser * Remove check_release * Release v0.1.1 * retain-test * fix-test * fix-lint --------- Co-authored-by: David Brochart <david.brochart@gmail.com>
Fixes y-crdt/pycrdt-websocket#116
Description
SQLite normally keeps freed pages in its internal freelist and only returns them to the OS when you rebuild the file. By turning on full auto-vacuum at database creation, any subsequent
DELETE
(including our TTL-based squash) will automatically release pages back to the OS—so the file will shrink as soon as old updates are purged.Code changes
Verification
logging number of rows and db file size