You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
zarrs automatically balances chunk and internal codec parallelism based on codec recommended_concurrency and the number of chunks involved in an operation. This works very well for sharding.
Some other codecs can support internal parallelism (e.g. zfp, blosc), but multithreading is intentionally disabled. Why? Because in my limited testing, the overhead seems quite high unless the chunks are massive, and I almost always use sharding with small inner chunks. It may be worth investigating these codecs to determine if it is worth kicking in codec parallelism (and with how many threads). The potential advantage is reduced memory usage (fewer chunks being decoded in parallel) and less cache thrashing.
The text was updated successfully, but these errors were encountered:
zarrs
is pretty quick (see the zarr_benchmarks repo), but maybe it can be even faster?zarrs
automatically balances chunk and internal codec parallelism based on codecrecommended_concurrency
and the number of chunks involved in an operation. This works very well for sharding.Some other codecs can support internal parallelism (e.g.
zfp
,blosc
), but multithreading is intentionally disabled. Why? Because in my limited testing, the overhead seems quite high unless the chunks are massive, and I almost always use sharding with small inner chunks. It may be worth investigating these codecs to determine if it is worth kicking in codec parallelism (and with how many threads). The potential advantage is reduced memory usage (fewer chunks being decoded in parallel) and less cache thrashing.The text was updated successfully, but these errors were encountered: