availability-recovery: move cpu burners in blocking tasks

[sandreim]

Fixes #7411

TODO:

• Address some availability data cloning issues in the impl
• Ensure chunks and backing group recovery works perfectly as before
• Versi burn-in analysis

[sandreim]

burning this in on Versi as part of #7378

[sandreim]

Versi burn-in shows an impressive CPU utilization of the reconstructed_data_matches_root (re-encode in the screenshot) with 2.5MB PoVs.

[burdges]

How long do these tasks take to finish?

It'd likely be better for cache locality if https://github.com/paritytech/reed-solomon-novelpoly itself were made multi-threaded, so then its tables could spend less time in L1. It's possible altering the tower of extension fields changes this somewhat, so maybe not the first thing to try, but otoh maybe easy, like just a rayon loop here

Update: L1 is per core, so this is not really true.

[sandreim]

How long do these tasks take to finish?

It'd likely be better for cache locality if https://github.com/paritytech/reed-solomon-novelpoly itself were made multi-threaded, so then its tables could spend less time in L1. It's possible altering the tower of extension fields changes this somewhat, so maybe not the first thing to try, but otoh maybe easy, like just a rayon loop here

Yes, pretty sure we can be faster here, but I expect that in practice the PoVs would rarely be this big (vs current synthetic Glutton test). This change set just makes sure we don't block other async tasks when reconstructing or re-encoding after data has been retrieved.

[burdges]

Reencode takes almost 50% longer than reconstruct. I guess reconstruct does some extra work, as reencode should produce 4x as much data.

We should ask soramitsu to look at the rust code, given their leopard fork claims to do this 2x faster.
https://github.com/soramitsu/erasure-coding-crust

Your benchmarks are for 2.5 megs yes? We're actually 4x faster than their claims, but I'm not really sure what their test setup is, like what they mean by VM.

[sandreim]

Reencode takes almost 50% longer than reconstruct. I guess reconstruct does some extra work, as reencode should produce 4x as much data.

We should ask soramitsu to look at the rust code, given their leopard fork claims to do this 2x faster. https://github.com/soramitsu/erasure-coding-crust

Your benchmarks are for 2.5 megs yes? We're actually 4x faster than their claims, but I'm not really sure what their test setup is, like what they mean by VM.

Yes, it is 2.5MiB, but we would need to be sure we have similar test env and input to claim that either is faster.

[eskimor]

Looks good overall.

Some thoughts:

1. How quick is the erasure encoding/decoding for very small PoVs ... could it be that the overhead of sending the data to a different thread is not worth it for those?
2. The architecture of sending to a different thread/task, while we don't have anything else to do is a bit counter intuitive at first - worth adding comments, describing what you are doing here.

[sandreim]

Looks good overall.

Some thoughts:

1. How quick is the erasure encoding/decoding for very small PoVs ... could it be that the overhead of sending the data to a different thread is not worth it for those?

Min value seems to be 96ms on some node (representative of 32K PoVs), max value is for 2.5MiB PoVs.

2. The architecture of sending to a different thread/task, while we don't have anything else to do is a bit counter intuitive at first - worth adding comments, describing what you are doing here.

Sure, will add some comments

[sandreim]

bot merge

[burdges]

Min value seems to be 96ms on some node (representative of 32K PoVs), max value is for 2.5MiB PoVs.

Is this reconstruction only? It's maybe the cost of building the walsh table. We tried to embed all the tables as constants, although initially we generated them at process start up, but you always need some table specific to the chunks form which your reconstructing.

Interestingly, this table could be reused if you've exactly the same chunks. I doubt this helps us, but maybe it buys you some orthogonality in paritytech/polkadot-sdk#607, although probably no.