Deduplicating queries when using multiple ModelMultipleChoiceFilter w/ same queryset callable

[joeyorlando]

hi there 👋

Chiming in with a question from the grafana/oncall open source project.

We use django_filter extensively in the project. I'm noticing some performance issues, which originate from our usage of a FilterSet subclass we've created. This is resulting in a lot of duplicate SQL queries:

Basically the problem is with these fields here where we specify a custom queryset callable. Each filter field class in AlertGroupFilter ends up calling these callables and recreating their own QuerySet objects, resulting in tons of duplicate queries. Ideally we’d be able to reuse these querysets somehow.. but I’m currently stumped on how this could be approached.

If I modify those callable methods to simply return <Model>.objects.none(), the duplicate queries disappear.

[carltongibson]

I did a talk on this topic for DjangoCon Europe 2020:

Choose, and choose quickly: Optimising ModelChoiceField: https://www.youtube.com/watch?v=e52S1SjuUeM

Give that a watch.

tl;dr If you set the choices explicitly (from a one-time cached value) you can avoid the repeated query overhead.

[joeyorlando]

Amazing! Going to check this out. Thanks 🙂

[joeyorlando]

This talk was very informative, especially the bit between the 14-15min mark.

I have a draft PR , w/ more context, of what I would assume should work based on my understanding of your DjangoCon talk, but it's not quite there.

A few other ideas I've brainstormed/tried:

1. a per-request/thread-safe LocMemCache cache, where I can cache the results of the functions passed in to the queryset arguments. I tried this but it's not (easily) possible to cache non-pickleable objects using LocMemCache.
2. getting rid of the filter fields in AlertGroupFilter that reference the get_user_queryset function, and somehow doing the filtering of all of these fields within some applicable method on the AlertGroupFilter (override the base filter_queryset?). This way I only need to call User.objects.filter(organization=request.user.organization).distinct() once within that method.