Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Delete access logs after specified time #5084

Closed
wants to merge 8 commits into from

Conversation

rgraber
Copy link

@rgraber rgraber commented Aug 28, 2024

Checklist

  1. If you've added code that should be tested, add tests
  2. If you've changed APIs, update (or create!) the documentation
  3. Ensure the tests pass
  4. Make sure that your code lints and that you've followed our coding style
  5. Write a title and, if necessary, a description of your work suitable for publishing in our release notes
  6. Mention any related issues in this repository (as #ISSUE) and in other repositories (as kobotoolbox/other#ISSUE)
  7. Open an issue in the docs if there are UI/UX changes

Description

Creates a background task to delete access logs after a number of days, specified in the Constance configuration as ACCESS_LOG_LIFESPAN.

Notes

Creates a spawning task that collects all ids of non-submission group access logs over ACCESS_LOG_LIFESPAN days old, and all submission groups where the latest date is more than ACCESS_LOG_LIFESPAN days ago, and enqueues a subtask per batch of 1000. The subtask then fetches the audit logs and deletes the batch.
The batching is to prevent tying up the database (or the celery queue) for too long, and so if something goes wrong with one batch of deletions the rest can continue.
The more-itertools library provides an easier way of breaking up the list of logs to be deleted into equal-sized sublists. It is a well-maintained library with a solid reputation https://github.com/more-itertools/more-itertools. The built-in itertools library actually has this method, but only in python 3.12. If and when we get there, we can remove the new import.

Blocked by #5080.

@rgraber rgraber changed the base branch from main to beta August 28, 2024 20:07
@rgraber rgraber marked this pull request as draft August 28, 2024 20:44
@rgraber rgraber changed the base branch from beta to beta-refactored August 29, 2024 14:32
@rgraber rgraber changed the base branch from beta-refactored to TASK-869-group-submissions August 29, 2024 20:29
@rgraber rgraber marked this pull request as ready for review August 30, 2024 13:04
@rgraber rgraber closed this Sep 3, 2024
@rgraber rgraber mentioned this pull request Sep 5, 2024
7 tasks
@rgraber
Copy link
Author

rgraber commented Sep 5, 2024

Replaced by #5097

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants