Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add option to disable cache #764

Open
pgulley opened this issue Aug 26, 2024 · 3 comments
Open

Add option to disable cache #764

pgulley opened this issue Aug 26, 2024 · 3 comments
Assignees

Comments

@pgulley
Copy link
Member

pgulley commented Aug 26, 2024

While testing the mediacloud stack, the cache sometimes gets in the way of being able to reproduce certain performance bugs. The ability to toggle it off (probably via an environment variable at launch) would go a long way to making this easier to test

@philbudne
Copy link
Contributor

At least for searches, a run-time checkbox (visible only to staff) would be helpful for monitoring historical ingest progress, and doing regular (5 minute?) tests on response time (put into grafana, and triggering alerts if searches fail or take too long).

I did a (VERY) quick look to see how search results are cached, and didn't see it in web-search (but I don't think I looked at the providers library).

I'd be happy to do the backend bits of this!

@philbudne
Copy link
Contributor

I made an attempt to allow disabling of caching in mc-providers here: mediacloud/mc-providers#32

@pgulley
Copy link
Member Author

pgulley commented Sep 16, 2024

Posted this in the providers#32, but repeating this here: https://stackoverflow.com/questions/77919265/django-can-i-force-a-request-to-bypass-the-cache-even-if-the-cache-key-already
It seems we could inject a custom version of the 'fetch-from-cache' middleware that would just let us bypass the cache on a given request, which would circumvent the issue where us double-dipping the overview query could confuse the cache. The example here looks for a flag in the requests headers rather than in the request body, which also strikes me as potentially cleaner

@github-staff github-staff deleted a comment from Lxx-c Oct 23, 2024
@github-staff github-staff deleted a comment from Lxx-c Oct 23, 2024
@github-staff github-staff deleted a comment from Lxx-c Oct 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants
@pgulley @philbudne @Evan-Leon and others