Speed up sliding sync by computing extensions in parallel

[erikjohnston]

The main change here is to add a helper function gather_optional_coroutines, which works in a similar way as yieldable_gather_results but takes a set of coroutines rather than a function

[MadLittleMods]

General idea looks good 👍

Surprised we need extra code

[MadLittleMods]

I'm surprised this sort of thing doesn't exist in the API? Like Promise.all(...) in JavaScript.

Is it because we can't use asyncio yet?

If it's just a feature-difference on how None is handled, can we change them from optional coroutines to Promise.resolve(None) to wrap it in a coroutine (however best to do that in Python) so we're always working with a coroutine?

[erikjohnston]

The reason we need to do this manually (rather than using built-ins), is that we need to correctly handle log contexts.

As a quick summary: basically whenever we "fork off" execution of some async code we need to use run_in_background (or run_as_background_process if we're not going to later wait on the result), and when we "join" them again we need to use make_deferred_yieldable. This is basically what gather_optional_coroutines is doing: spawning the coroutines as background tasks, and then calling make_deferred_yieldable when waiting on them all to complete

[MadLittleMods]

This context would be useful to read in the docstrings. I see log contexts are mentioned but none of the consequences or why we need to or why we care is included.