Merge branch 'main' into devnet-3

Resolve conflict: keep both is_aggregator field and pending_block_parents
from main's storage refactor.

Author: MegaRedHand

.github/prompts/ai-review.md

@@ -18,3 +18,8 @@ Consensus-layer considerations:
 
 Be concise and specific. Provide line references when suggesting changes.
 If the code looks good, acknowledge it briefly.
+
+Formatting rules:
+- NEVER use `#N` (e.g. #1, #2) for enumeration — GitHub renders those as issue/PR references. Use `1.`, `2.`, etc. or bullet points instead.
+- When referring back to items, use "Item 1", "Point 2", etc. — never "Issue #1" or "#1".Collapse comment
+

.github/workflows/ci.yml

@@ -47,12 +47,71 @@ jobs:
     steps:
       - uses: actions/checkout@v6
 
-      - name: Download test fixtures
-        env:
-          GH_TOKEN: ${{ github.token }}
+      # Read the pinned leanSpec commit from the Makefile (single source of truth)
+      - name: Get leanSpec pinned commit
+        id: lean-spec
+        run: echo "commit=$(sed -n 's/^LEAN_SPEC_COMMIT_HASH:= *//p' Makefile)" >> $GITHUB_OUTPUT
+
+      - name: Cache test fixtures
+        id: cache-fixtures
+        uses: actions/cache@v4
+        with:
+          path: leanSpec/fixtures
+          key: leanspec-fixtures-${{ steps.lean-spec.outputs.commit }}
+
+      # All fixture generation steps are skipped when the cache hits
+      - name: Checkout leanSpec at pinned commit
+        if: steps.cache-fixtures.outputs.cache-hit != 'true'
+        uses: actions/checkout@v6
+        with:
+          repository: leanEthereum/leanSpec
+          ref: ${{ steps.lean-spec.outputs.commit }}
+          path: leanSpec
+
+      - name: Install uv and Python 3.14
+        if: steps.cache-fixtures.outputs.cache-hit != 'true'
+        uses: astral-sh/setup-uv@v4
+        with:
+          enable-cache: true
+          cache-dependency-glob: "leanSpec/pyproject.toml"
+          python-version: "3.14"
+
+      - name: Sync leanSpec dependencies
+        if: steps.cache-fixtures.outputs.cache-hit != 'true'
+        working-directory: leanSpec
+        run: uv sync --no-progress
+
+      - name: Get production keys URL hash
+        if: steps.cache-fixtures.outputs.cache-hit != 'true'
+        id: prod-keys-url
+        working-directory: leanSpec
         run: |
-          mkdir -p leanSpec/fixtures
-          gh run download --repo leanEthereum/leanSpec --name fixtures-prod-scheme --dir leanSpec/fixtures
+          URL=$(uv run python -c "from consensus_testing.keys import KEY_DOWNLOAD_URLS; print(KEY_DOWNLOAD_URLS['prod'])")
+          HASH=$(echo -n "$URL" | sha256sum | awk '{print $1}')
+          echo "hash=$HASH" >> $GITHUB_OUTPUT
+
+      - name: Cache production keys
+        if: steps.cache-fixtures.outputs.cache-hit != 'true'
+        id: cache-prod-keys
+        uses: actions/cache@v4
+        with:
+          path: leanSpec/packages/testing/src/consensus_testing/test_keys/prod_scheme
+          key: prod-keys-${{ steps.prod-keys-url.outputs.hash }}
+
+      - name: Download production keys
+        if: steps.cache-fixtures.outputs.cache-hit != 'true' && steps.cache-prod-keys.outputs.cache-hit != 'true'
+        working-directory: leanSpec
+        run: uv run python -m consensus_testing.keys --download --scheme prod
+
+      - name: Generate test fixtures
+        if: steps.cache-fixtures.outputs.cache-hit != 'true'
+        working-directory: leanSpec
+        run: uv run fill --fork=Devnet --scheme prod -o fixtures -n 2
+
+      # Ensure make sees fixtures as up-to-date (its timestamp must be
+      # newer than leanSpec/, which intermediate steps may have modified).
+      - name: Mark fixtures as up-to-date
+        run: touch leanSpec/fixtures
 
       - name: Setup Rust
         uses: dtolnay/rust-toolchain@master

.gitignore

@@ -30,3 +30,9 @@ target
 
 # Used for generating test vectors
 /leanSpec
+
+# Personal Claude Code overrides (not shared with team)
+CLAUDE.local.md
+
+# Log output by make run-devnet
+devnet.log

CLAUDE.md

@@ -13,20 +13,28 @@ Not to be confused with Ethereum consensus clients AKA Beacon Chain clients AKA
 
 ```
 bin/ethlambda/              # Entry point, CLI, orchestration
+  └─ src/version.rs         # Build-time version info (vergen-git2)
 crates/
   blockchain/               # State machine actor (GenServer pattern)
     ├─ src/lib.rs           # BlockChain actor, tick events, validator duties
     ├─ src/store.rs         # Fork choice store, block/attestation processing
+    ├─ src/key_manager.rs   # Validator key management and signing
+    ├─ src/metrics.rs       # Blockchain-level Prometheus metrics
     ├─ fork_choice/         # LMD GHOST implementation (3SF-mini)
     └─ state_transition/    # STF: process_slots, process_block, attestations
+        └─ src/metrics.rs   # State transition timing + counters
   common/
     ├─ types/               # Core types (State, Block, Attestation, Checkpoint)
     ├─ crypto/              # XMSS aggregation (leansig wrapper)
-    └─ metrics/             # Prometheus metrics
+    └─ metrics/             # Prometheus re-exports, TimingGuard, gather utilities
   net/
     ├─ p2p/                 # libp2p: gossipsub + req-resp (Status, BlocksByRoot)
-    └─ rpc/                 # Axum HTTP endpoints (/lean/v0/* and /metrics)
+    │   ├─ src/gossipsub/   # Topic encoding, message handling
+    │   ├─ src/req_resp/    # Request/response codec and handlers
+    │   └─ src/metrics.rs   # Peer connection/disconnection tracking
+    └─ rpc/                 # Axum HTTP: /lean/v0/{states,checkpoints,health} + /metrics
   storage/                  # RocksDB backend, in-memory for tests
+    └─ src/api/             # StorageBackend trait + Table enum
 ```
 
 ## Key Architecture Patterns
@@ -64,7 +72,7 @@ Fork choice head update
 
 ### Before Committing
 ```bash
-cargo fmt                                    # Format code
+make fmt                                     # Format code (cargo fmt --all)
 make lint                                    # Clippy with -D warnings
 make test                                    # All tests + forkchoice (with skip-signature-verification)
 ```
@@ -73,6 +81,8 @@ make test                                    # All tests + forkchoice (with skip
 ```bash
 .claude/skills/test-pr-devnet/scripts/test-branch.sh    # Test branch in multi-client devnet
 rm -rf leanSpec && make leanSpec/fixtures                # Regenerate test fixtures (requires uv)
+make docker-build                                        # Build Docker image (DOCKER_TAG=local)
+make run-devnet                                          # Run local devnet with lean-quickstart
 ```
 
 ### Testing with Local Devnet
@@ -95,10 +105,10 @@ let byte: u8 = code.into();
 ### Ownership for Large Structures
 ```rust
 // Prefer taking ownership to avoid cloning large data (signatures ~3KB)
-pub fn consume_signed_block(signed_block: SignedBlockWithAttestation) { ... }
+pub fn insert_signed_block(&mut self, root: H256, signed_block: SignedBlockWithAttestation) { ... }
 
 // Add .clone() at call site if needed - makes cost explicit
-store.insert_signed_block(root, signed_block.clone());
+store.insert_signed_block(block_root, signed_block.clone());
 ```
 
 ### Formatting Patterns
@@ -157,12 +167,34 @@ let result = operation()
     .map_err(|err| CustomError::from(err))?;
 ```
 
-### Metrics (RAII Pattern)
+### Metrics Patterns
+
+**Registration with `LazyLock`:**
+```rust
+// Module-scoped statics (preferred for state_transition metrics)
+static LEAN_STATE_TRANSITION_TIME_SECONDS: LazyLock<Histogram> = LazyLock::new(|| {
+    register_histogram!("lean_metric_name", "Description", vec![...]).unwrap()
+});
+
+// Function-scoped statics (used in blockchain metrics)
+pub fn update_head_slot(slot: u64) {
+    static LEAN_HEAD_SLOT: LazyLock<IntGauge> = LazyLock::new(|| {
+        register_int_gauge!("lean_head_slot", "Latest slot").unwrap()
+    });
+    LEAN_HEAD_SLOT.set(slot.try_into().unwrap());
+}
+```
+
+**RAII timing guard (auto-observes duration on drop):**
 ```rust
-// Timing guard automatically observes duration on drop
 let _timing = metrics::time_state_transition();
 ```
 
+**All metrics use `ethlambda_metrics::*` re-exports** — the `ethlambda-metrics` crate re-exports
+prometheus types (`IntGauge`, `IntCounter`, `Histogram`, etc.) and provides `TimingGuard` + `gather_default_metrics()`.
+
+**Naming convention:** All metrics use `lean_` prefix (e.g., `lean_head_slot`, `lean_state_transition_time_seconds`).
+
 ### Logging Patterns
 
 **Use tracing shorthand syntax for cleaner logs:**
@@ -281,8 +313,27 @@ cargo test -p ethlambda-blockchain --features skip-signature-verification --test
 - Crypto tests marked `#[ignore]` (slow leanVM operations)
 
 ### Storage Architecture
-- Genesis block has no signatures - stored in Blocks table only, not BlockSignatures
-- All other blocks must have entries in both tables
+- Blocks are split into three tables: `BlockHeaders`, `BlockBodies`, `BlockSignatures`
+- Genesis/anchor blocks have empty bodies (detected via `EMPTY_BODY_ROOT`) — no entry in `BlockBodies`
+- Genesis block has no signatures — no entry in `BlockSignatures`
+- All other blocks must have entries in all three tables
+- `LiveChain` table provides fast `(slot||root) → parent_root` index for fork choice
+- Storage uses trait-based API: `StorageBackend` → `StorageReadView` (reads) + `StorageWriteBatch` (atomic writes)
+
+### Storage Tables (10)
+
+| Table | Key → Value | Purpose |
+|-------|-------------|---------|
+| `BlockHeaders` | H256 → BlockHeader | Block headers by root |
+| `BlockBodies` | H256 → BlockBody | Block bodies (empty for genesis) |
+| `BlockSignatures` | H256 → BlockSignaturesWithAttestation | Signatures (absent for genesis) |
+| `States` | H256 → State | Beacon states by root |
+| `LatestKnownAttestations` | u64 → AttestationData | Fork-choice-active attestations |
+| `LatestNewAttestations` | u64 → AttestationData | Pending (pre-promotion) attestations |
+| `GossipSignatures` | SignatureKey → ValidatorSignature | Individual validator signatures |
+| `AggregatedPayloads` | SignatureKey → Vec\<AggregatedSignatureProof\> | Aggregated proofs |
+| `Metadata` | string → various | Store state (head, config, checkpoints) |
+| `LiveChain` | (slot\|\|root) → parent\_root | Fast fork choice traversal index |
 
 ### State Root Computation
 - Always computed via `tree_hash_root()` after full state transition
@@ -304,6 +355,7 @@ cargo test -p ethlambda-blockchain --features skip-signature-verification --test
 - `tree_hash`: Merkle tree hashing
 - `spawned-concurrency`: Actor model
 - `libp2p`: P2P networking (custom LambdaClass fork)
+- `vergen-git2`: Build-time git commit/branch info embedded in binary
 
 **Storage:**
 - `rocksdb`: Persistent backend
@@ -313,6 +365,7 @@ cargo test -p ethlambda-blockchain --features skip-signature-verification --test
 
 **Specs:** `leanSpec/src/lean_spec/` (Python reference implementation)
 **Devnet:** `lean-quickstart` (github.com/blockblaz/lean-quickstart)
+**Releases:** See `RELEASE.md` for release process documentation
 
 ## Other implementations
 

Makefile

@@ -1,8 +1,11 @@
-.PHONY: help lint docker-build run-devnet test
+.PHONY: help fmt lint docker-build run-devnet test
 
 help: ## 📚 Show help for each of the Makefile recipes
 	@grep -E '^[a-zA-Z0-9_-]+:.*?## .*$$' $(MAKEFILE_LIST) | sort | awk 'BEGIN {FS = ":.*?## "}; {printf "\033[36m%-30s\033[0m %s\n", $$1, $$2}'
 
+fmt: ## 🎨 Format all code using rustfmt
+	cargo fmt --all
+
 lint: ## 🔍 Run clippy on all workspace crates
 	cargo clippy --workspace --all-targets -- -D warnings
 
@@ -20,6 +23,7 @@ docker-build: ## 🐳 Build the Docker image
 		--build-arg GIT_COMMIT=$(GIT_COMMIT) \
 		--build-arg GIT_BRANCH=$(GIT_BRANCH) \
 		-t ghcr.io/lambdaclass/ethlambda:$(DOCKER_TAG) .
+	@echo
 
 LEAN_SPEC_COMMIT_HASH:=b39472e73f8a7d603cc13d14426eed14c6eff6f1
 
@@ -33,12 +37,14 @@ leanSpec/fixtures: leanSpec
 lean-quickstart:
 	git clone https://github.com/blockblaz/lean-quickstart.git --depth 1 --single-branch
 
-
-# TODO: start metrics too
 run-devnet: docker-build lean-quickstart ## 🚀 Run a local devnet using lean-quickstart
-	# Go to lean-quickstart/local-devnet/genesis/validator-config.yaml to modify
-	# the validator configuration for the local devnet.
-	# NOTE: to run the local image of ethlambda, make sure to set the image tag
-	# in lean-quickstart/client-cmds/ethlambda-cmd.sh to "ghcr.io/lambdaclass/ethlambda:local"
-	cd lean-quickstart \
-	&& NETWORK_DIR=local-devnet ./spin-node.sh --node all --generateGenesis --metrics
+	@echo "Starting local devnet with ethlambda client (\"$(DOCKER_TAG)\" tag). Logs will be dumped in devnet.log, and metrics served in http://localhost:3000"
+	@echo
+	@echo "Devnet will be using the current configuration. For custom configurations, modify lean-quickstart/local-devnet/genesis/validator-config.yaml and restart the devnet."
+	@echo
+	@# Use temp file instead of sed -i for macOS/GNU portability
+	@sed 's|ghcr.io/lambdaclass/ethlambda:[^ ]*|ghcr.io/lambdaclass/ethlambda:$(DOCKER_TAG)|' lean-quickstart/client-cmds/ethlambda-cmd.sh > lean-quickstart/client-cmds/ethlambda-cmd.sh.tmp \
+		&& mv lean-quickstart/client-cmds/ethlambda-cmd.sh.tmp lean-quickstart/client-cmds/ethlambda-cmd.sh
+	@echo "Starting local devnet. Press Ctrl+C to stop all nodes."
+	@cd lean-quickstart \
+		&& NETWORK_DIR=local-devnet ./spin-node.sh --node all --generateGenesis --metrics > ../devnet.log 2>&1

README.md

@@ -4,10 +4,17 @@ Minimalist, fast and modular implementation of the Lean Ethereum client written
 
 ## Getting started
 
-We use `cargo` as our build system. To build and run the client, simply run:
+We use `cargo` as our build system, but prefer `make` as a convenient wrapper for common tasks. These are some common targets:
 
 ```sh
-cargo run
+# Formats all code
+make fmt
+# Checks and lints the code
+make lint
+# Runs all tests
+make test
+# Builds a docker image tagged as "ghcr.io/lambdaclass/ethlambda:local"
+make docker-build DOCKER_TAG=local
 ```
 
 Run `make help` or take a look at our [`Makefile`](./Makefile) for other useful commands.
@@ -21,9 +28,11 @@ To run a local devnet with multiple clients using [lean-quickstart](https://gith
 make run-devnet
 ```
 
-This generates fresh genesis files and starts all three clients with metrics enabled.
+This generates fresh genesis files and starts all configured clients with metrics enabled.
 Press `Ctrl+C` to stop all nodes.
 
+For custom devnet configurations, go to `lean-quickstart/local-devnet/genesis/validator-config.yaml` and edit the file before running the command above. See `lean-quickstart`'s documentation for more details on how to configure the devnet.
+
 ## Philosophy
 
 Many long-established clients accumulate bloat over time. This often occurs due to the need to support legacy features for existing users or through attempts to implement overly ambitious software. The result is often complex, difficult-to-maintain, and error-prone systems.
@@ -68,7 +77,11 @@ Additional features:
 
 ### pq-devnet-2
 
-We support the [pq-devnet-2 spec](https://github.com/leanEthereum/pm/blob/main/breakout-rooms/leanConsensus/pq-interop/pq-devnet-2.md). A Docker tag `devnet2` is available for this version.
+Support for the [pq-devnet-2 spec](https://github.com/leanEthereum/pm/blob/main/breakout-rooms/leanConsensus/pq-interop/pq-devnet-2.md) will be ending soon (see ["older devnets"](#older-devnets)). A Docker tag `devnet2` is currently available for this version.
+
+### pq-devnet-3
+
+We are working on adding support for the [pq-devnet-3 spec](https://github.com/leanEthereum/pm/blob/main/breakout-rooms/leanConsensus/pq-interop/pq-devnet-3.md). A Docker tag `devnet3` will be published for this version.
 
 ### Older devnets
 
@@ -80,6 +93,10 @@ Support for older devnet releases is discontinued when the next devnet version i
 
 Some features we are looking to implement in the near future, in order of priority:
 
-- Checkpoint sync for long-lived networks
-- Observability: more metrics from leanMetrics and better logs
-- RPC endpoints for chain data consumption
+- [Checkpoint sync for long-lived networks](https://github.com/lambdaclass/ethlambda/issues/80)
+- [pq-devnet-3 support](https://github.com/lambdaclass/ethlambda/issues/73)
+- [Fetching of unknown blocks referenced by attestations](https://github.com/lambdaclass/ethlambda/issues/91)
+- [Discarding blocks with invalid signatures](https://github.com/lambdaclass/ethlambda/issues/78)
+- [Non-finalization hardening: bound memory usage during long periods of non-finalization](https://github.com/lambdaclass/ethlambda/issues/103)
+- [Observability: more metrics from leanMetrics](https://github.com/lambdaclass/ethlambda/issues/76)
+- [RPC endpoints for chain data consumption](https://github.com/lambdaclass/ethlambda/issues/75)