Add data migration to purge deleted submissions #744
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
WARNING: may be slow on large databases. Mention this in release notes. Closes #733.
noliveleger
reviewed
Jun 30, 2021
Comment on lines
+18
to
+20
| print( | ||
| f'Purging {to_purge.count()} deleted instances...', end='', flush=True | ||
| ) |
There was a problem hiding this comment.
FYI: I cannot see the output of these print .
In other migrations, we used sys.stdout.write and sys.stderr.write
There was a problem hiding this comment.
Oops, I'm only now seeing this! I'm surprised you couldn't see it the output if count was greater than zero on your database, but print() certainly doesn't seem ideal to use here (although neither does sys.stdout.write 😉). I guess migrations are part of a management command and theoretically ought to have access somehow to self.stdout provided somewhere by the BaseCommand class.
Be more insistent that I acknowledge your feedback next time 😛
noliveleger
approved these changes
Jun 30, 2021
This was referenced Aug 5, 2021
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Removes
Instances in PostgreSQL wheredeleted_atis non-null and corresponding documents in MongoDB (whose_deleted_atattribute is non-null).Description
Automatically removes submissions that were already marked as deleted. Necessary for self-hosters so that the changes in #732 do not cause deleted submissions to resurface.
⚠️ Includes a migration that could be slow on large databases. If you run your own instance of KoBoToolbox with many thousands of submissions, consider using
manage.py shell_plusinside the KoBoCAT container (kobo-install/run.py -cf exec kobocat bash) to purge deleted submissions using the code shown in #696 prior to upgrading to this release.Related issues
Closes #733