Reconstruct data columns without blocking processing and import #5990

jimmygchen · 2024-06-25T04:56:07Z

Issue Addressed

Part of #4983. This is a work in progress.

The current sequence of reconstruction:

Receive blocks & data columns
As soon as the supernode receives 50% of columns, start reconstruction (while holding the availability cache write lock)
After reconstruction completes, the node imports block and publishes the remaining 50% of columns
Now it may also receives the remaining 50%, which gets verified and then ignored

Proposed Changes

Changes to data column reconstruction:
- Attempt reconstruction without holding availability cache lock, so we can process other gossip / rpc data columns simultaneously
- Check availability cache again before publishing reconstructed columns to avoid publishing excess duplicates
Remove some unnecessary RuntimeVariableList conversion for data columns

Additional Info

Below are logs from local testing, notice that the reconstruction blocks processing of incoming data columns (due to holding the availability cache write lock), and they gets ignored as duplicates:

Jun 24 08:01:15.781 DEBG Successfully verified gossip data column sidecar, index: 63, block_root: 0x690d…0e3e, slot: 40556
Jun 24 08:01:16.280 DEBG Reconstructed columns                   count: 64, service: availability_cache, service: beacon
Jun 24 08:01:16.281 DEBG Writing data_columns to store           count: 128, block_root: 0x690d…0e3e, service: beacon
Jun 24 08:01:16.284 INFO Gossipsub data column processed, imported fully available block, block_root: 0x690d…0e3e
Jun 24 08:01:16.284 DEBG Sending pubsub messages                 topics: [DataColumnSidecar(DataColumnSubnetId(0)), DataColumnSidecar(DataColumnSubnetId(1)), DataColumnSidecar(DataColumnSubnetId(2)) .. DataColumnSidecar(DataColumnSubnetId(31))], count: 64, service: network
Jun 24 08:01:16.314 DEBG Successfully verified gossip data column sidecar, index: 64, block_root: 0x690d…0e3e, slot: 40556
Jun 24 08:01:16.314 DEBG Ignoring gossip column already imported, data_column_index: 64, block_root: 0x690d4885fe843d085464b4f36b774597525d2dbddcd26ceca95fd4fdf4930e3e
Jun 24 08:01:16.323 DEBG Successfully verified gossip data column sidecar, index: 65, block_root: 0x690d…0e3e, slot: 40556
Jun 24 08:01:16.323 DEBG Ignoring gossip column already imported, data_column_index: 65, block_root: 0x690d4885fe843d085464b4f36b774597525d2dbddcd26ceca95fd4fdf4930e3e
Jun 24 08:01:16.332 DEBG Successfully verified gossip data column sidecar, index: 66, block_root: 0x690d…0e3e, slot: 40556
Jun 24 08:01:16.332 DEBG Ignoring gossip column already imported, data_column_index: 66, block_root: 0x690d4885fe843d085464b4f36b774597525d2dbddcd26ceca95fd4fdf4930e3e
Jun 24 08:01:16.341 DEBG Successfully verified gossip data column sidecar, index: 67, block_root: 0x690d…0e3e, slot: 40556
Jun 24 08:01:16.341 DEBG Ignoring gossip column already imported, data_column_index: 67, block_root: 0x690d4885fe843d085464b4f36b774597525d2dbddcd26ceca95fd4fdf4930e3e
Jun 24 08:01:16.349 DEBG Successfully verified gossip data column sidecar, index: 68, block_root: 0x690d…0e3e, slot: 40556
Jun 24 08:01:16.349 DEBG Ignoring gossip column already imported, data_column_index: 68, block_root: 0x690d4885fe843d085464b4f36b774597525d2dbddcd26ceca95fd4fdf4930e3e
Jun 24 08:01:16.358 DEBG Successfully verified gossip data column sidecar, index: 69, block_root: 0x690d…0e3e, slot: 40556
Jun 24 08:01:16.358 DEBG Ignoring gossip column already imported, data_column_index: 69, block_root: 0x690d4885fe843d085464b4f36b774597525d2dbddcd26ceca95fd4fdf4930e3e
Jun 24 08:01:16.367 DEBG Successfully verified gossip data column sidecar, index: 70, block_root: 0x690d…0e3e, slot: 40556

mergify · 2024-06-25T04:56:44Z

⚠️ The sha of the head commit of this PR conflicts with #5986. Mergify cannot evaluate rules on this PR. ⚠️

beacon_node/beacon_chain/src/block_verification_types.rs

beacon_node/beacon_chain/src/data_availability_checker/overflow_lru_cache.rs

beacon_node/network/src/network_beacon_processor/mod.rs

…ng one at a time. Address other review comments.

jimmygchen · 2024-06-28T04:43:18Z

Thanks for the review! I've addressed the comments and will start local testing.

beacon_node/network/src/network_beacon_processor/sync_methods.rs

beacon_node/network/src/network_beacon_processor/gossip_methods.rs

beacon_node/beacon_chain/src/data_availability_checker/overflow_lru_cache.rs

…d metrics.

jimmygchen · 2024-07-01T03:45:18Z

Thanks for the review! Addressed your comments in 7d2d826.

Squashed commit of the following: commit 7d2d826 Author: Jimmy Chen <jchen.tc@gmail.com> Date: Mon Jul 1 13:44:45 2024 +1000 Send import results to sync after reconstruction. Add more logging and metrics. commit 4b30ebe Merge: f93e2b5 7206909 Author: Jimmy Chen <jchen.tc@gmail.com> Date: Fri Jun 28 17:23:22 2024 +1000 Merge branch 'das' into fork/reconstruct-without-blocking-import commit f93e2b5 Author: Jimmy Chen <jchen.tc@gmail.com> Date: Fri Jun 28 14:42:04 2024 +1000 Code cleanup: add type aliases and update comments. commit 6ac055d Author: Jimmy Chen <jchen.tc@gmail.com> Date: Fri Jun 28 14:26:40 2024 +1000 Revert reconstruction behaviour to always go ahead rather than allowing one at a time. Address other review comments. commit 1e3964e Author: Jimmy Chen <jchen.tc@gmail.com> Date: Tue Jun 25 00:02:19 2024 +1000 Reconstruct columns without blocking processing and import.

jimmygchen · 2024-07-01T06:40:05Z

beacon_node/network/src/network_beacon_processor/gossip_methods.rs

@@ -1044,7 +1017,7 @@ impl<T: BeaconChainTypes> NetworkBeaconProcessor<T> {
                            "block_root" => %block_root,
                        );

-                        // Potentially trigger reconstruction
+                        self.attempt_data_column_reconstruction(block_root).await;


Looks like this doesn't work, issues:

Still seeing reconstruction blocking gossip data column processing

This change should be made to the cached block instead of the clone:

lighthouse/beacon_node/beacon_chain/src/data_availability_checker/overflow_lru_cache.rs

Line 885 in 7d2d826

pending_components.reconstruction_started();

Because of 1., after the block is imported, I see a bunch of:

Jul 01 06:27:21.130 DEBG Attempted to publish duplicate message kind: data_column_sidecar_31, service: libp2p

Fixed #2.

I can't seem to figure out why reconstruction is blocking the processing of other incoming data columns. I don't see any lock being held during reconstruction, perhaps it is just waiting for a beacon processor worker to become available.

We could send it to the back of the beacon processor queue, but I'll do some more testing to make sure this is the case.

…of the clone.

dapplion · 2024-08-16T08:43:56Z

Generally looks good to me! Lots of conflicts, tho

Do you want to port this to unstable?

jimmygchen · 2024-08-19T12:41:56Z

Thanks, I've just tried to resolve conflicts, but unfortunately I see this is going to conflict a lot with #6268, and it might be easier to implement it once that one is merged, as it has quite a bit of common logic and changes. Since this one is very outdated, I think it make sense to just create a new PR, using the branch from #6268 as base branch.

jimmygchen · 2024-08-22T12:18:35Z

Leaving this PR open so we don't forget this.

michaelsproul · 2024-08-27T05:20:31Z

Please update to point at unstable by either:

Rebasing on unstable (if your branch has a small number of commits that are easy to tease out), or
Merging origin/das into this PR: git fetch origin; git merge origin/das. This will result in the smallest number of conflict resolutions and is better for branches that already contain merge commits or have extensive history.

jimmygchen · 2024-09-17T04:35:30Z

Closing in favour of #6403

Reconstruct columns without blocking processing and import.

1e3964e

jimmygchen mentioned this pull request Jun 25, 2024

Reconstruct data columns without blocking processing and import #5986

Closed

2 tasks

jimmygchen added work-in-progress PR is a work-in-progress das Data Availability Sampling labels Jun 25, 2024

dapplion requested changes Jun 25, 2024

View reviewed changes

jimmygchen self-assigned this Jun 26, 2024

jimmygchen added 2 commits June 28, 2024 14:26

Revert reconstruction behaviour to always go ahead rather than allowi…

6ac055d

…ng one at a time. Address other review comments.

Code cleanup: add type aliases and update comments.

f93e2b5

jimmygchen added ready-for-review The code is ready for review and removed work-in-progress PR is a work-in-progress labels Jun 28, 2024

jimmygchen requested a review from dapplion June 28, 2024 04:43

Merge branch 'das' into fork/reconstruct-without-blocking-import

4b30ebe

dapplion requested changes Jun 28, 2024

View reviewed changes

Send import results to sync after reconstruction. Add more logging an…

7d2d826

…d metrics.

jimmygchen requested a review from dapplion July 1, 2024 03:45

jimmygchen added the das-devnet-1 label Jul 1, 2024

jimmygchen mentioned this pull request Jul 1, 2024

peerdas-devnet-1 testing branch #6023

Closed

Downgrade noisy log.

0f355a7

jimmygchen force-pushed the reconstruct-without-blocking-import branch from db43a2a to 0f355a7 Compare July 1, 2024 05:57

jimmygchen commented Jul 1, 2024

View reviewed changes

jimmygchen added work-in-progress PR is a work-in-progress and removed ready-for-review The code is ready for review labels Jul 1, 2024

jimmygchen force-pushed the reconstruct-without-blocking-import branch from 808e84a to 3badc3b Compare July 11, 2024 04:41

Update reconstruction status on PendingComponents from cache instead …

efabb98

…of the clone.

jimmygchen force-pushed the reconstruct-without-blocking-import branch from 3badc3b to efabb98 Compare July 11, 2024 04:53

jimmygchen mentioned this pull request Jul 16, 2024

DAS - Tracking Issue #4983

Open

52 tasks

jimmygchen added ready-for-review The code is ready for review and removed work-in-progress PR is a work-in-progress labels Aug 16, 2024

jimmygchen closed this Aug 19, 2024

jimmygchen added do-not-merge and removed ready-for-review The code is ready for review labels Aug 22, 2024

jimmygchen reopened this Aug 22, 2024

jimmygchen mentioned this pull request Aug 27, 2024

PeerDAS implementation #5683

Merged

mergify bot deleted the branch sigp:das August 27, 2024 04:10

mergify bot closed this Aug 27, 2024

dapplion reopened this Aug 28, 2024

jimmygchen mentioned this pull request Sep 17, 2024

Refactor data column reconstruction and avoid blocking processing #6403

Merged

jimmygchen closed this Sep 17, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Reconstruct data columns without blocking processing and import #5990

Reconstruct data columns without blocking processing and import #5990

jimmygchen commented Jun 25, 2024 •

edited

Loading

mergify bot commented Jun 25, 2024

jimmygchen commented Jun 28, 2024

jimmygchen commented Jul 1, 2024

jimmygchen Jul 1, 2024 •

edited by dapplion

Loading

jimmygchen Jul 11, 2024

dapplion commented Aug 16, 2024

jimmygchen commented Aug 19, 2024 •

edited

Loading

jimmygchen commented Aug 22, 2024

michaelsproul commented Aug 27, 2024

jimmygchen commented Sep 17, 2024

Reconstruct data columns without blocking processing and import #5990

Reconstruct data columns without blocking processing and import #5990

Conversation

jimmygchen commented Jun 25, 2024 • edited Loading

Issue Addressed

Proposed Changes

Additional Info

mergify bot commented Jun 25, 2024

jimmygchen commented Jun 28, 2024

jimmygchen commented Jul 1, 2024

jimmygchen Jul 1, 2024 • edited by dapplion Loading

Choose a reason for hiding this comment

jimmygchen Jul 11, 2024

Choose a reason for hiding this comment

dapplion commented Aug 16, 2024

jimmygchen commented Aug 19, 2024 • edited Loading

jimmygchen commented Aug 22, 2024

michaelsproul commented Aug 27, 2024

jimmygchen commented Sep 17, 2024

jimmygchen commented Jun 25, 2024 •

edited

Loading

jimmygchen Jul 1, 2024 •

edited by dapplion

Loading

jimmygchen commented Aug 19, 2024 •

edited

Loading