Fix high memory usage bug with chunked HK relationship #1333

[timothymillar]

• Fixes High memory usage when calculating chunked Hamilton-Kerr relationship matrix #1333
• Tests added
• User visible changes (including notable bug fixes) are documented in changelog.rst

[jeromekelleher]

LGTM!

Is there a time-cost for rechunking like this on small inputs? (I'm not entirely clear what happens by calling rechuink)

[timothymillar]

I don't think there is any cost to this rechunk unless the 'chunks' argument is actually used. By default, this will just 'rechunk' a single chunk array to another single chunk array. Dask doesn't appear to even include the operation in the task graph (with .visualize(optimize_graph=False)).

I tried rechunking a dummy array to its original size 10_000 times in a loop and don't notice any slowdown on .compute(). So I'm assuming that this case is recognized at call time and doesn't affect the actual compute graph.

[timothymillar]

Looks like the CI failures are unrelated, Docs failing on 3.11 isn't ideal but it seems to be an issue with the ipynb tutorials not executing.