Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Delete expired access logs #5097

Draft
wants to merge 21 commits into
base: TASK-869-group-submissions-better
Choose a base branch
from

Conversation

rgraber
Copy link

@rgraber rgraber commented Sep 5, 2024

Checklist

  1. If you've added code that should be tested, add tests
  2. If you've changed APIs, update (or create!) the documentation
  3. Ensure the tests pass
  4. Make sure that your code lints and that you've followed our coding style
  5. Write a title and, if necessary, a description of your work suitable for publishing in our release notes
  6. Mention any related issues in this repository (as #ISSUE) and in other repositories (as kobotoolbox/other#ISSUE)
  7. Open an issue in the docs if there are UI/UX changes

Description

Delete access logs older than the configured number of days (defaults to 60). The number of days is configurable in constance.

Notes

Adds a new task to run every day at midnight.

  1. Figure out the cut off date based on the configured lifespan of the access logs.
  2. Identify all submission logs older than the cut off date and remove them from their submission groups
  3. Collect all empty submission groups
  4. In batches, submit the empty submission groups for deletion by spawning new batch_delete_audit_logs_by_id tasks
  5. Collect all other access logs older than the cut off date
  6. In batches, submit these for deletion in the same way

Additional changes:
Add a queryset to the SubmissionAccessLog proxy model to make finding all submission logs easier.
Add the more-itertools library so we can easily break up the list of ids into batches of a constant size. Plain itertools has this functionality in python 3.12, but for the older versions we need the more-itertools library.

The batch size is configurable in settings. It defaults to 1000.

Blocked by #5094
Replaces #5084

@rgraber rgraber marked this pull request as ready for review September 5, 2024 17:59
@rgraber rgraber force-pushed the TASK-869-group-submissions-better branch from 67579ff to 55b66ea Compare September 17, 2024 12:47
@rgraber rgraber marked this pull request as draft September 18, 2024 19:32
@rgraber
Copy link
Author

rgraber commented Sep 18, 2024

converting to draft until the questions about grouping submissions are resolved

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant