Merge branch '__develop' into electra-better-packing

# Conflicts:
#	CHANGELOG.md

CHANGELOG.md

@@ -4,7 +4,81 @@ All notable changes to this project will be documented in this file.
 
 The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.
 
-## [Unreleased](https://github.com/prysmaticlabs/prysm/compare/v5.1.0...HEAD)
+## [Unreleased](https://github.com/prysmaticlabs/prysm/compare/v5.1.2...HEAD)
+
+### Added
+
+- Electra EIP6110: Queue deposit [pr](https://github.com/prysmaticlabs/prysm/pull/14430)
+- Add Bellatrix tests for light client functions.
+- Add Discovery Rebooter Feature.
+- Added GetBlockAttestationsV2 endpoint.
+- Light client support: Consensus types for Electra
+- Added SubmitPoolAttesterSlashingV2 endpoint.
+- Added SubmitAggregateAndProofsRequestV2 endpoint.
+- Updated the `beacon-chain/monitor` package to Electra. [PR](https://github.com/prysmaticlabs/prysm/pull/14562)
+- Better attestation packing for Electra. [PR](https://github.com/prysmaticlabs/prysm/pull/14534)
+
+### Changed
+
+- Electra EIP6110: Queue deposit requests changes from consensus spec pr #3818
+- reversed the boolean return on `BatchVerifyDepositsSignatures`, from need verification, to all keys successfully verified
+- Fix `engine_exchangeCapabilities` implementation.
+- Updated the default `scrape-interval` in `Client-stats` to 2 minutes to accommodate Beaconcha.in API rate limits.
+- Switch to compounding when consolidating with source==target.
+- Revert block db save when saving state fails.
+- Return false from HasBlock if the block is being synced. 
+- Cleanup forkchoice on failed insertions.
+- Use read only validator for core processing to avoid unnecessary copying.
+- Use ROBlock across block processing pipeline
+
+### Deprecated
+
+- `/eth/v1alpha1/validator/activation/stream` grpc wait for activation stream is deprecated. [pr](https://github.com/prysmaticlabs/prysm/pull/14514)
+
+### Removed
+
+- Removed finalized validator index cache, no longer needed.
+
+### Fixed
+
+- Fixed mesh size by appending `gParams.Dhi = gossipSubDhi`
+- Fix skipping partial withdrawals count.
+- wait for the async StreamEvent writer to exit before leaving the http handler, avoiding race condition panics [pr](https://github.com/prysmaticlabs/prysm/pull/14557)
+- Certain deb files were returning a 404 which made building new docker images without an existing
+  cache impossible. This has been fixed with updates to rules_oci and bazel-lib.
+
+### Security
+
+## [v5.1.2](https://github.com/prysmaticlabs/prysm/compare/v5.1.1...v5.1.2) - 2024-10-16 
+
+This is a hotfix release with one change. 
+
+Prysm v5.1.1 contains an updated implementation of the beacon api streaming events endpoint. This
+new implementation contains a bug that can cause a panic in certain conditions. The issue is
+difficult to reproduce reliably and we are still trying to determine the root cause, but in the
+meantime we are issuing a patch that recovers from the panic to prevent the node from crashing.
+
+This only impacts the v5.1.1 release beacon api event stream endpoints. This endpoint is used by the
+prysm REST mode validator (a feature which requires the validator to be configured to use the beacon
+api intead of prysm's stock grpc endpoints) or accessory software that connects to the events api,
+like https://github.com/ethpandaops/ethereum-metrics-exporter
+
+### Fixed 
+
+- Recover from panics when writing the event stream [#14545](https://github.com/prysmaticlabs/prysm/pull/14545)
+
+## [v5.1.1](https://github.com/prysmaticlabs/prysm/compare/v5.1.0...v5.1.1) - 2024-10-15
+
+This release has a number of features and improvements. Most notably, the feature flag 
+`--enable-experimental-state` has been flipped to "opt out" via `--disable-experimental-state`. 
+The experimental state management design has shown significant improvements in memory usage at
+runtime. Updates to libp2p's gossipsub have some bandwidith stability improvements with support for
+IDONTWANT control messages. 
+
+The gRPC gateway has been deprecated from Prysm in this release. If you need JSON data, consider the
+standardized beacon-APIs. 
+
+Updating to this release is recommended at your convenience.
 
 ### Added
 
@@ -18,15 +92,14 @@ The format is based on Keep a Changelog, and this project adheres to Semantic Ve
 - GetBeaconStateV2: add Electra case.
 - Implement [consensus-specs/3875](https://github.com/ethereum/consensus-specs/pull/3875).
 - Tests to ensure sepolia config matches the official upstream yaml.
+- `engine_newPayloadV4`,`engine_getPayloadV4` used for electra payload communication with execution client.  [pr](https://github.com/prysmaticlabs/prysm/pull/14492)
 - HTTP endpoint for PublishBlobs.
 - GetBlockV2, GetBlindedBlock, ProduceBlockV2, ProduceBlockV3: add Electra case.
 - Add Electra support and tests for light client functions.
 - fastssz version bump (better error messages).
 - SSE implementation that sheds stuck clients. [pr](https://github.com/prysmaticlabs/prysm/pull/14413)
-- Add Bellatrix tests for light client functions.
-- Add Discovery Rebooter Feature.
-- Added GetBlockAttestationsV2 endpoint.
-- Better attestation packing for Electra. [PR](https://github.com/prysmaticlabs/prysm/pull/14534)
+- Added GetPoolAttesterSlashingsV2 endpoint.
+- Use engine API get-blobs for block subscriber to reduce block import latency and potentially reduce bandwidth.
 
 ### Changed
 
@@ -56,17 +129,18 @@ The format is based on Keep a Changelog, and this project adheres to Semantic Ve
 - Updated Sepolia bootnodes.
 - Make committee aware packing the default by deprecating `--enable-committee-aware-packing`.
 - Moved `ConvertKzgCommitmentToVersionedHash` to the `primitives` package.
+- Updated correlation penalty for EIP-7251. 
 
 ### Deprecated
 - `--disable-grpc-gateway` flag is deprecated due to grpc gateway removal.
 - `--enable-experimental-state` flag is deprecated. This feature is now on by default. Opt-out with `--disable-experimental-state`.
-- `/eth/v1alpha1/validator/activation/stream` grpc wait for activation stream is deprecated. [pr](https://github.com/prysmaticlabs/prysm/pull/14514)
 
 ### Removed
 
-- removed gRPC Gateway
-- Removed unused blobs bundle cache
+- Removed gRPC Gateway.
+- Removed unused blobs bundle cache.
 - Removed consolidation signing domain from params. The Electra design changed such that EL handles consolidation signature verification.
+- Remove engine_getPayloadBodiesBy{Hash|Range}V2
 
 ### Fixed
 
@@ -85,10 +159,11 @@ The format is based on Keep a Changelog, and this project adheres to Semantic Ve
 - Light client support: fix light client attested header execution fields' wrong version bug.
 - Testing: added custom matcher for better push settings testing.
 - Registered `GetDepositSnapshot` Beacon API endpoint.
-- Fixed mesh size by appending `gParams.Dhi = gossipSubDhi`
 
 ### Security
 
+No notable security updates.
+
 ## [v5.1.0](https://github.com/prysmaticlabs/prysm/compare/v5.0.4...v5.1.0) - 2024-08-20
 
 This release contains 171 new changes and many of these are related to Electra! Along side the Electra changes, there

WORKSPACE

@@ -101,9 +101,9 @@ http_archive(
 
 http_archive(
     name = "aspect_bazel_lib",
-    sha256 = "f5ea76682b209cc0bd90d0f5a3b26d2f7a6a2885f0c5f615e72913f4805dbb0d",
-    strip_prefix = "bazel-lib-2.5.0",
-    url = "https://github.com/aspect-build/bazel-lib/releases/download/v2.5.0/bazel-lib-v2.5.0.tar.gz",
+    sha256 = "a272d79bb0ac6b6965aa199b1f84333413452e87f043b53eca7f347a23a478e8",
+    strip_prefix = "bazel-lib-2.9.3",
+    url = "https://github.com/bazel-contrib/bazel-lib/releases/download/v2.9.3/bazel-lib-v2.9.3.tar.gz",
 )
 
 load("@aspect_bazel_lib//lib:repositories.bzl", "aspect_bazel_lib_dependencies", "aspect_bazel_lib_register_toolchains")
@@ -165,7 +165,7 @@ load("@rules_oci//oci:pull.bzl", "oci_pull")
 oci_pull(
     name = "linux_debian11_multiarch_base",  # Debian bullseye
     digest = "sha256:b82f113425c5b5c714151aaacd8039bc141821cdcd3c65202d42bdf9c43ae60b",  # 2023-12-12
-    image = "gcr.io/distroless/cc-debian11",
+    image = "gcr.io/prysmaticlabs/distroless/cc-debian11",
     platforms = [
         "linux/amd64",
         "linux/arm64/v8",
@@ -227,7 +227,7 @@ filegroup(
     url = "https://github.com/ethereum/EIPs/archive/5480440fe51742ed23342b68cf106cefd427e39d.tar.gz",
 )
 
-consensus_spec_version = "v1.5.0-alpha.6"
+consensus_spec_version = "v1.5.0-alpha.8"
 
 bls_test_version = "v0.1.1"
 
@@ -243,7 +243,7 @@ filegroup(
     visibility = ["//visibility:public"],
 )
     """,
-    integrity = "sha256-M7u/Ot/Vzorww+dFbHp0cxLyM2mezJjijCzq+LY3uvs=",
+    integrity = "sha256-BsGIbEyJuYrzhShGl0tHhR4lP5Qwno8R3k8a6YBR/DA=",
     url = "https://github.com/ethereum/consensus-spec-tests/releases/download/%s/general.tar.gz" % consensus_spec_version,
 )
 
@@ -259,7 +259,7 @@ filegroup(
     visibility = ["//visibility:public"],
 )
     """,
-    integrity = "sha256-deOSeLRsmHXvkRp8n2bs3HXdkGUJWWqu8KFM/QABbZg=",
+    integrity = "sha256-DkdvhPP2KiqUOpwFXQIFDCWCwsUDIC/xhTBD+TZevm0=",
     url = "https://github.com/ethereum/consensus-spec-tests/releases/download/%s/minimal.tar.gz" % consensus_spec_version,
 )
 
@@ -275,7 +275,7 @@ filegroup(
     visibility = ["//visibility:public"],
 )
     """,
-    integrity = "sha256-Zz7YCf6XVf57nzSEGq9ToflJFHM0lAGwhd18l9Rf3hA=",
+    integrity = "sha256-vkZqV0HB8A2Uc56C1Us/p5G57iaHL+zw2No93Xt6M/4=",
     url = "https://github.com/ethereum/consensus-spec-tests/releases/download/%s/mainnet.tar.gz" % consensus_spec_version,
 )
 
@@ -290,7 +290,7 @@ filegroup(
     visibility = ["//visibility:public"],
 )
     """,
-    integrity = "sha256-BoXckDxXnDcEmAjg/dQgf/tLiJsb6CT0aZvmWHFijrY=",
+    integrity = "sha256-D/HPAW61lKqjoWwl7N0XvhdX+67dCEFAy8JxVzqBGtU=",
     strip_prefix = "consensus-specs-" + consensus_spec_version[1:],
     url = "https://github.com/ethereum/consensus-specs/archive/refs/tags/%s.tar.gz" % consensus_spec_version,
 )

api/server/structs/conversions.go

@@ -1476,12 +1476,15 @@ func DepositSnapshotFromConsensus(ds *eth.DepositSnapshot) *DepositSnapshot {
 	}
 }
 
-func PendingBalanceDepositsFromConsensus(ds []*eth.PendingBalanceDeposit) []*PendingBalanceDeposit {
-	deposits := make([]*PendingBalanceDeposit, len(ds))
+func PendingDepositsFromConsensus(ds []*eth.PendingDeposit) []*PendingDeposit {
+	deposits := make([]*PendingDeposit, len(ds))
 	for i, d := range ds {
-		deposits[i] = &PendingBalanceDeposit{
-			Index:  fmt.Sprintf("%d", d.Index),
-			Amount: fmt.Sprintf("%d", d.Amount),
+		deposits[i] = &PendingDeposit{
+			Pubkey:                hexutil.Encode(d.PublicKey),
+			WithdrawalCredentials: hexutil.Encode(d.WithdrawalCredentials),
+			Amount:                fmt.Sprintf("%d", d.Amount),
+			Signature:             hexutil.Encode(d.Signature),
+			Slot:                  fmt.Sprintf("%d", d.Slot),
 		}
 	}
 	return deposits

api/server/structs/conversions_state.go

@@ -722,7 +722,7 @@ func BeaconStateElectraFromConsensus(st beaconState.BeaconState) (*BeaconStateEl
 	if err != nil {
 		return nil, err
 	}
-	pbd, err := st.PendingBalanceDeposits()
+	pbd, err := st.PendingDeposits()
 	if err != nil {
 		return nil, err
 	}
@@ -770,7 +770,7 @@ func BeaconStateElectraFromConsensus(st beaconState.BeaconState) (*BeaconStateEl
 		EarliestExitEpoch:             fmt.Sprintf("%d", eee),
 		ConsolidationBalanceToConsume: fmt.Sprintf("%d", cbtc),
 		EarliestConsolidationEpoch:    fmt.Sprintf("%d", ece),
-		PendingBalanceDeposits:        PendingBalanceDepositsFromConsensus(pbd),
+		PendingDeposits:               PendingDepositsFromConsensus(pbd),
 		PendingPartialWithdrawals:     PendingPartialWithdrawalsFromConsensus(ppw),
 		PendingConsolidations:         PendingConsolidationsFromConsensus(pc),
 	}, nil

api/server/structs/endpoints_beacon.go

@@ -176,7 +176,8 @@ type BLSToExecutionChangesPoolResponse struct {
 }
 
 type GetAttesterSlashingsResponse struct {
-	Data []*AttesterSlashing `json:"data"`
+	Version string          `json:"version,omitempty"`
+	Data    json.RawMessage `json:"data"` // Accepts both `[]*AttesterSlashing` and `[]*AttesterSlashingElectra` types
 }
 
 type GetProposerSlashingsResponse struct {

api/server/structs/endpoints_validator.go

@@ -15,7 +15,7 @@ type SubmitContributionAndProofsRequest struct {
 }
 
 type SubmitAggregateAndProofsRequest struct {
-	Data []*SignedAggregateAttestationAndProof `json:"data"`
+	Data []json.RawMessage `json:"data"`
 }
 
 type SubmitSyncCommitteeSubscriptionsRequest struct {

api/server/structs/other.go

@@ -257,9 +257,12 @@ type ConsolidationRequest struct {
 	TargetPubkey  string `json:"target_pubkey"`
 }
 
-type PendingBalanceDeposit struct {
-	Index  string `json:"index"`
-	Amount string `json:"amount"`
+type PendingDeposit struct {
+	Pubkey                string `json:"pubkey"`
+	WithdrawalCredentials string `json:"withdrawal_credentials"`
+	Amount                string `json:"amount"`
+	Signature             string `json:"signature"`
+	Slot                  string `json:"slot"`
 }
 
 type PendingPartialWithdrawal struct {

api/server/structs/state.go

@@ -176,7 +176,7 @@ type BeaconStateElectra struct {
 	EarliestExitEpoch             string                         `json:"earliest_exit_epoch"`
 	ConsolidationBalanceToConsume string                         `json:"consolidation_balance_to_consume"`
 	EarliestConsolidationEpoch    string                         `json:"earliest_consolidation_epoch"`
-	PendingBalanceDeposits        []*PendingBalanceDeposit       `json:"pending_balance_deposits"`
+	PendingDeposits               []*PendingDeposit              `json:"pending_deposits"`
 	PendingPartialWithdrawals     []*PendingPartialWithdrawal    `json:"pending_partial_withdrawals"`
 	PendingConsolidations         []*PendingConsolidation        `json:"pending_consolidations"`
 }

beacon-chain/blockchain/chain_info.go

@@ -12,6 +12,7 @@ import (
 	"github.com/prysmaticlabs/prysm/v5/beacon-chain/state"
 	fieldparams "github.com/prysmaticlabs/prysm/v5/config/fieldparams"
 	"github.com/prysmaticlabs/prysm/v5/config/params"
+	consensus_blocks "github.com/prysmaticlabs/prysm/v5/consensus-types/blocks"
 	"github.com/prysmaticlabs/prysm/v5/consensus-types/forkchoice"
 	"github.com/prysmaticlabs/prysm/v5/consensus-types/interfaces"
 	"github.com/prysmaticlabs/prysm/v5/consensus-types/primitives"
@@ -44,7 +45,7 @@ type ForkchoiceFetcher interface {
 	UpdateHead(context.Context, primitives.Slot)
 	HighestReceivedBlockSlot() primitives.Slot
 	ReceivedBlocksLastEpoch() (uint64, error)
-	InsertNode(context.Context, state.BeaconState, [32]byte) error
+	InsertNode(context.Context, state.BeaconState, consensus_blocks.ROBlock) error
 	ForkChoiceDump(context.Context) (*forkchoice.Dump, error)
 	NewSlot(context.Context, primitives.Slot) error
 	ProposerBoost() [32]byte

beacon-chain/blockchain/chain_info_forkchoice.go

@@ -4,6 +4,7 @@ import (
 	"context"
 
 	"github.com/prysmaticlabs/prysm/v5/beacon-chain/state"
+	consensus_blocks "github.com/prysmaticlabs/prysm/v5/consensus-types/blocks"
 	"github.com/prysmaticlabs/prysm/v5/consensus-types/forkchoice"
 	"github.com/prysmaticlabs/prysm/v5/consensus-types/primitives"
 )
@@ -44,10 +45,10 @@ func (s *Service) ReceivedBlocksLastEpoch() (uint64, error) {
 }
 
 // InsertNode is a wrapper for node insertion which is self locked
-func (s *Service) InsertNode(ctx context.Context, st state.BeaconState, root [32]byte) error {
+func (s *Service) InsertNode(ctx context.Context, st state.BeaconState, block consensus_blocks.ROBlock) error {
 	s.cfg.ForkChoiceStore.Lock()
 	defer s.cfg.ForkChoiceStore.Unlock()
-	return s.cfg.ForkChoiceStore.InsertNode(ctx, st, root)
+	return s.cfg.ForkChoiceStore.InsertNode(ctx, st, block)
 }
 
 // ForkChoiceDump returns the corresponding value from forkchoice