-
-
Notifications
You must be signed in to change notification settings - Fork 386
Lodestar Weekly Standups 2025
Phil Ngo edited this page Aug 1, 2025
·
32 revisions
The Lodestar team hosts planning/standup meetings weekly on Tuesdays at 3:00pm Universal Standard Time. These meetings allow the team to conduct release planning, prioritise tasks, sync on current issues, implementations and provide status updates on the Lodestar roadmap.
Note that these notes are transcribed and summarized by AI language models and may not accurately reflect the context discussed.
Agenda: https://github.com/ChainSafe/lodestar/discussions/8081
Recording/Transcript: https://drive.google.com/drive/folders/1rmMOalE_iS73jB-_roHvrEOSNOIsNfeR?usp=sharing
- DevNet 3 Launch Status: DevNet 3 has successfully launched with generally minimal problems reported
- However, significant syncing issues have emerged as the primary concern, particularly affecting the team's super nodes
- Issue #8093 - Genesis Validator Handling: The team discussed a technical issue brought up by Barnabas regarding how Genesis validators are considered as finalized state. The core problem centers around the custody group count (CGC) that only updates on finalized blocks. Nico proposed an alternative approach where CGC updates could trigger on every validator registration rather than waiting for finalized state, sparking technical debate about the proper implementation according to the Ethereum consensus specifications.
- Syncing Infrastructure Problems: The most critical issue identified involves three super nodes and Erigon-2 experiencing synchronization problems. Matthew reported that the super nodes are not properly connecting to super node peers, with the root cause still under investigation. This represents a significant operational challenge that requires immediate attention.
- Benchmarks posted here seem to indicate much better performance with bun-ffi: https://github.com/ChainSafe/lodestar/issues/7280#issuecomment-3113254649
- Getting Bun to work with beacon-node is a priority and we're keeping track of external blockers in this issue: https://github.com/ChainSafe/lodestar/issues/7280
- LevelDB NAPI Binding is erroring: If you require the binding from the main thread and then you require the same binding from a worker thread, there is an error instead of resolving the module properly.
- The issue described in the Lodestar standup refers to a well-documented Node.js problem where native modules with NAPI bindings fail to load properly when required in worker threads. This is a common issue that affects many native Node.js modules, including the classic-level (LevelDB) database module that Lodestar uses.
-
Classic-level Database Module Issues: A complex technical problem was identified with the classic-level database module when used in worker thread contexts:
- The classic-level module uses native C++ bindings through NAPI to interface with the LevelDB database. When a native module is loaded in the main thread and then subsequently loaded in a worker thread, Node.js can encounter several problems:
- "Module did not self-register" Error: This is the most common error message that appears when native modules fail to properly initialize in worker thread contexts.
- Context Awareness Problem: Native modules need to be "context-aware" to work properly across multiple V8 contexts (main thread vs worker threads). Many older native modules were not designed with this requirement in mind.
- Binding Singleton Issues: As mentioned in the transcript, there are "two pointers that need to get shared" - the native database pointer and the bindings singleton. These don't properly transfer between main and worker processes.
- Cayman discovered that while the module functions properly in the main process, it fails when required within worker threads. The issue appears to stem from module loading and caching mechanisms not properly handling the transition between main and worker processes.
- Issue posted here: https://github.com/oven-sh/bun/issues/21432
- The technical discussion revealed that the problem involves two critical pointers that need proper sharing: the native database pointer and the bindings singleton. Multiple team members (Cayman, Matthew, Nazar) are collaborating to understand why the same bindings aren't being loaded consistently across main and child processes
- Currently a blocker for using level. We will consider using SQLite or something else that's more stable.
- Issue with being able to use bun types as it relies on react types as a peer dependency.
- Seems like they use it for experimental types
- We are in favor of creating an issue and reference it into the exception list we have
- Hashtree now has a generic C fallback: https://github.com/OffchainLabs/hashtree/pull/43/files
- We will clone from a commit hash
- Migration to pnpm starting
- Once dependency issues are complete, we will move onto runtime issues for Bun
- Tuyen continuing with testing
process_epochfunction. Having sanity tests there continue to help catch bugs. - Bing continuing on the main state transition function and breaking out PR into smaller PRs.
- Additional work was completed on QUIC bindings with Grapebaba (Kai) and connecting it with the larger zig-libp2p implementation
- Grapebaba focused on writing the QUIC transport tests, Had issues troubleshooting this week potentially due to differences between Zig/C/C++ compilers, but upstreamed some issues with lsquic. Temporarily bypassed by forking repo.
Database and Repository Refactor:
- Drafted and submitted a PR for changes to the database repositories.
- Waiting for Matthew to review this initial PR.
- Focused on understanding and implementing updates to the SSZ types, specifically around encoding and decoding, to improve familiarity and map out next steps.
Backfill Mechanism Research:
- Investigated how Prysm's codebase handles backfill:
- Noted support for batch-based backfillβfetching backfill data in batches of one epoch (32 slots per batch).
- Observed that Prysm provides configuration flags for backfill behavior, such as batch size and the ability to parallelize backfilling using multiple worker threads.
- Discovered that backfill in Prysm is still in a developmental stage and is disabled by default.
Comparisons with Other Clients:
- Began reviewing Lighthouse and Teku implementation strategies:
- Noted that TEKU seemingly fetches blocks iteratively (one-by-one), but indicated uncertainty and plans to confirm details.
Next Steps and Workflow:
- Working on a main backfill class redesign and plans to draft a PR with this new design.
- Intends to create a walkthrough video to explain this work when ready.
- Mentioned this week's progress was slower due to spending time learning aspects of Golang and other client codebases, but expects to realign and catch up next week.
- Optimized Pool Scanning: Tuyen discussed improvements in handling attestations, focusing on reducing redundant scans. After a new attestation is processed, there is typically no need to rescan already-seen attestations in the pool. This leads to a much smaller and more efficient attestation pool.
- Mainnet Performance: When running a node that subscribes to subnets on mainnet, Tuyen observed effective consolidation of attestations, with 98β99% of cases reaching the expected "package" (i.e., group) size for attestations. The impact on block profitability was minimal or only slightly positive, indicating the optimization is safe and practical for production use.
- Future Improvements: Tuyen mentioned that, in the future, once the proposer assignments for the next epoch are known, the node can preemptively subscribe for those proposers. This would maintain the same efficiency, further improving the attestation handling pipeline.
- Extensive Unit Testing: Significant effort was devoted to building comprehensive unit tests for the new code. The last step before full completion is to confirm that a regular (non-validator) node can successfully run the implementation, produce a block, and see it included on-chain. Tuyen hopes to achieve this milestone in the near-term (possibly by slot 123).
- Proposal Lookahead Enhancements: Tuyen made improvements in "proposal lookahead" by introducing a final process for epoch handling, addressing differences between epoch zero ("fu") and later epochs to make the logic more straightforward.
- Benchmarking NAPI vs. FFI: Tuyen ran benchmarks comparing native code integrations (FFI vs. NAPI bindings, e.g., for hash functions). Results indicated that NAPI performance was actually worse than not using it, particularly for hash operations. Since FFI lacks async support, this highlighted a need to revisit and re-benchmark using the current module (possibly BSD).
Key Takeaways
- Attestation pool management is more efficient, reducing CPU/scan overhead.
- Mainnet testing confirms practical effectiveness and negligible downside risk.
- Additional architectural work on epoch transitions and performance benchmarking is ongoing.
- Extensive unit tests and practical block production verification are the remaining steps before full rollout of this optimization.
libp2p Issue (Related to Nimbus Interop)
- Context: Matthew brought up a networking problem initially highlighted by Nazar and previously discussed with Cayman. The issue centers on Lodestar's interaction with the libp2p networking stackβspecifically with the Nimbus client.
- Symptoms: The error appears during "recrest" (request) methods and manifests primarily in communication with Nimbus, rather than with other Ethereum consensus clients.
- Nature of the Problem: The behavior is "tricky" and affects timing-sensitive methods. The root cause remains unclear, and Cayman and Matthew have both been investigating. Matthew acknowledged it's showing up across different request pathways, suggesting it's a protocol/interop bugβnot isolated to a single API or message type.
- Status: The issue is under active investigation, with Cayman also providing metrics/monitoring improvements to help diagnose the problem. The team intends to add further metrics to better isolate and understand the problem from the main thread's perspective.
Cooloff Period PR
- Progress: Matthew is also working on updating a pull request regarding the "cooloff period" logic, incorporating a second (or third) round of review comments from Tian.
- Next Steps: He plans to push these changes shortly, with the expectation there may be another round of comments. He is committed to closing out this review cycle promptly.
Matthew
- Actively investigating super node connectivity issues in DevNet 3
- Debugging why super nodes aren't connecting to designated peers
- Contributing technical expertise to the classic-level worker thread problem analysis
- Taking ownership of complex infrastructure debugging tasks
Nico
- Conducted research into issue #8093 regarding Genesis validator CGC handling
- Proposed alternative technical approach for CGC updates based on validator registration events
- Challenged existing implementation assumptions about finalized state requirements
- Bringing fresh perspective to established technical patterns
Cayman
- Identified and documented specific technical issue with classic-level module in worker threads
- Confirmed reproducible failure pattern: works in main process, fails in worker context
- Leading investigation into module loading and caching problems
- Providing detailed technical analysis of Node.js worker thread behaviors
Nazar
- Contributing technical analysis on worker thread binding issues
- Questioning fundamental assumptions about binding consistency across processes
- Helping debug native database pointer sharing problems
- Bringing systems-level expertise to complex threading issues
NC
- Completed read-through of EPBS spec and corrected inconsistencies with PRs
- Attestation Deadline Timing Refactor: Noted an outstanding issue previously opened regarding the migration from the use of "intervals per slot" to "milliseconds" for attestation deadline calculations and sync committees within Lodestar. This change is significant because both the EPBS specification and ongoing "shorter time slot" network research depend on precise time-based deadlines instead of slot-based intervals.
- Now sees this as an optimal moment to implement the change in Lodestar, with work already in progress and anticipated to be completed within a few days (noting the change itself is relatively small in scope).
Agenda: https://github.com/ChainSafe/lodestar/discussions/8068
Recording/Transcript: https://drive.google.com/drive/folders/1ICiSIUrCw4rVDaz_h5O6Mq_EKEWgo-BI?usp=sharing
-
The team identified critical issues with networking performance related to PeerDAS implementation.
-
The primary issue involves column overlap causing excessive requests to the same peers, resulting in rate limiting and disconnections - Matt explained that the sync struggles stem from how range sync requests (reqresps) are happening, leading to "hammering the same peers for lots of batches" because of column overlap.
-
This issue causes nodes to get downscored and disconnected from peers, requiring a cooldown period before they can reconnect and resume syncing.
-
The same fundamental problem affects genesis sync, where nodes encounter rate limiting after approximately 1,000 slots and must wait again.
-
Tuyen is actively working on implementing rate limiting solutions to prevent constantly hitting the same peers and getting banned.
-
The fix involves reducing the download requirement from 128 columns to 64 columns for super node configurations, though the issue affects both super nodes and regular nodes.
-
fusaka-devnet-2has been stabilized and all systems are running smoothly. - The network is now fully operational after resolving the peer connectivity and rate limiting issues [2][3].
-
fusaka-devnet-3preparation is progressing well with only an update required for https://github.com/ethereum/builder-specs/pull/123. - Matthew noted that DevNet 3 should be smooth since it's essentially a spec freeze running the same code to ensure it passes all BPO forks without problems.
- The team expects DevNet 3 to launch soon, with three PRs remaining to be merged before launch
- Merged feature into unstable branch: https://github.com/ChainSafe/lodestar/pull/7810
- Testing can be done in Kurtosis with artificial block delays to see if it is properly reorging weak heads, then we can determine whether to enable it by default
- Small issue found and fixed in a follow up PR: https://github.com/ChainSafe/lodestar/pull/8079
A significant portion of the discussion centered around Antithesis-style deterministic simulation testing. Cayman shared insights from a conversation about Tiger Beetle's development approach, which started with building a simulation framework first, then creating a database that could hook into their simulation.
Key points about deterministic simulation testing:
- Tiger Beetle was created by first building a simulation framework for transactions, then building a database that could integrate with their simulation
- This approach enables deterministic testing where you can "change time" and replay scenarios exactly
- Antithesis provides deterministic testing capabilities that have been successfully used for blockchain testing, including extensive use during the Ethereum Merge
- The framework acts like a "time machine" where you can recreate any problem down to the second
The team expressed strong interest in exploring this approach for Lodestar, with Cayman noting it's "definitely worth looking into" as an "interesting paradigm of programming" for their new initiatives.
Nena is coordinating with Matthew on integrating Kurtosis testing with Lodestar's existing simulation tests.
The plan involves:
- Understanding the differences between what Kurtosis currently covers versus Lodestar's assertion coverage
- Determining how to bring both frameworks together effectively
- Figuring out session management and data retrieval mechanisms (HTTP RPC vs hooks)
- Successfully stabilized all networks with comprehensive fixes for the sync and peer management issues
- DevNet 2 is now fully operational with all nodes running smoothly
- DevNet 3 is ready for deployment with the same stable codebase
- Blacklist PR has gone through another round of review comments and has been addressed, with only timing considerations remaining
- Completed work on the rec rest PR that was pending, moving changes from the "puras branch to unstable"
- Making progress on the by-root sync implementation with updated unit tests planned for completion this week
- Matthew confirmed that getting everything from the PeerDAS branch onto unstable is "the heart of the issue".
- Cayman provided excellent review comments on the by-root sync implementation
- Matthew has started addressing these with additional unit tests planned.
Agenda: https://github.com/ChainSafe/lodestar/discussions/8050
Recording/Transcript: https://drive.google.com/drive/folders/1N953MUSYSTDbjBH5WheVfua58vxjdN8V?usp=sharing
- The team discussed the readiness of Lodestar 1.32, which has been in beta for six days.
- Phil confirmed the team was ready to release later that day, noting improved performance metrics from recent updates.
- The release includes important optimizations for block production, parallelization of block building steps, and builder/engine race improvements[see release notes].
-
DevNet 2 Status: The team reported that Fusaka DevNet 2 is running fine with some debugging work ongoing.
-
Matt mentioned debugging a memory issue brought up by Barnabas and completing blacklisting work
-
Peering issues with Nimbus client peering were noted, requiring further investigation with their team[1].
-
DevNet 3 Preparation: Two key PRs remain for DevNet 3:
- Nico's ENR PR (needs fork digest function updates) and the submit blinded block v2 PR (awaiting spec discussions on status codes).
- The team is preparing for DevNet 3 launch expected later in July.
The team discussed the upcoming Glamsterdam hardfork headliner selection, with the decision scheduled for July 24, 2025. Key topics included:
- EPBS (Enshrined Proposer-Builder Separation): Strong consensus in the Fusaka breakout room supporting EPBS as the headliner, with FOCIL as complementary
- Slot Timing: Discussion of whether to pursue 6-second slots or maintain 12-second slots with EPBS implementation
- PureETH Advocacy: Cayman suggested pushing for PureETH as the execution layer headliner to complement EPBS and Fusaka
- Timeline considerations were discussed, with the team noting that shipping Fusaka before DevConnect seems impossible, with mainnet potentially targeting March 2026.
- The team discussed their trial with an AI code review bot.
- Initial feedback indicated that 80% of suggestions were documentation-related, which was considered noise.
- The team decided to configure the bot to ignore documentation issues and focus on logic errors and bugs, extending the trial for another month.
- The team decided to also include the bot for some of the Zig libraries being worked on.
Matt
- Memory Issues: Debugged memory issues and completed memory profiling work
- Blacklisting Implementation: Finished blacklisting work using peerscore delay/decay approach
- Backfill Research: Spent time with Katya reviewing backfill implementations, examining Teku codebase and preparing for Lighthouse research
- DevNet 3 Preparation: Noted minimal PeerDAS changes needed for DevNet 3
- Nimbus Issues: Planning to investigate libp2p protocol negotiation issues specific to Nimbus client
Katya
- Dashboard Updates: Updated PeerDAS dashboard with reconstruction metrics for DevNet 3 comparison
- Backfill Research: Collaborated with Matthew on backfill strategy, examining Teku and Lighthouse codebases
- Nimbus Engine Issues: Identified missing responses from Nimbus on engine_getBlobsV2, requiring investigation
- Metrics Work: Preparing FOCIL metrics implementation for the Fusaka group's evaluation needs
NC
- DevNet 3 PRs: Two key PRs remain - ENR PR needing fork digest updates and submit blinded block v2 PR
- Fusaka Support: Confirmed EPBS + Fusaka combination is popular in breakout rooms and technically feasible
- FOCIL Development: Working on FOCIL branch rebasing and preparing for spec updates
- SSZ Implementation: Implementing JSON functions in Zig with bug fixes already merged
Nico
- ENR PR: Needs to update PR due to fork digest function renaming
- Block Production: Working on block packing profitability testing and single attestation implementation
- Fusaka DevNet 2: Debugging issues resolved, major differences now on unstable branch
- Performance Testing: Conducting proposal boost profitability analysis
Tuyen
- State Transition in Zig: Achieved 90-95% port completion of TypeScript state transition to Zig
- PeerDAS Work: Implemented data column subnet piece availability to unblock subscription optimizations
- Single Attestation: Planning to work on block production flow integration
- Testing Status: Working to stabilize Zig implementation for broader team collaboration[1]
Cayman
- QUIC Integration: Spent several days working on QUIC with Lodestar, encountering rate limiting, garbled responses, and uninitialized data issues
- Zig Fork Choice: Continuing work on Zig fork choice implementation
- LibP2P Upgrade: Discussed necessity of upgrading from year-old LibP2P version due to security and maintenance concerns
Bing
- Zig Libraries: Reviewing state transition implementation and smaller library issues
- Beam Chain: Recently discovered beam chain post-quantum devnet requirements involving QUIC with Zig LibP2P
- Issue Resolution: Opening issues and fixing configuration problems in SSZ and other libraries
- Grapebaba: Working on Zig LibP2P proposal for GossipSub and QUIC, validating C API calls and preparing specification draft
- Nena: Researching Lodestar's sim test framework for Kurtosis integration, preparing requirements documentation
- Vedant: Exploring backfill implementations across clients, benchmarking performance, developing algorithm from scratch in coordination with Matthew and Katya
- Bun Runtime: Cayman reported challenges with LibP2P compatibility in Bun, requiring monkey patching and significant debugging
- State Transition: Tuyen's Zig implementation showing strong progress despite current instability
- Timeline: Estimated 3-month timeline for Lodestar 2.0 completion, potentially targeting DevConnect for major announcements
Agenda: https://github.com/ChainSafe/lodestar/discussions/7926
Recording/Transcript: https://drive.google.com/drive/folders/1N953MUSYSTDbjBH5WheVfua58vxjdN8V?usp=sharing
- Chiemerie raised concerns about implementing log-based metrics for attestation monitoring. The infrastructure team prefers using built-in metrics for alerting rather than parsing logs, as log-based queries can be expensive and potentially slow down the Loki instance.
- Log-based queries can be expensive because they require scanning through large volumes of log data. This is particularly problematic in blockchain environments where:
- Validators generate continuous streams of attestation-related logs
- Log volumes can be substantial with hundreds of thousands of active validators
- Complex pattern matching requires computational resources that can "slow down the Loki instance"
- Suggested integration of a collection point in the metrics codebase and we can alert on the metric as opposed to current strategy of parsing through collected logs
- The team agreed to follow up with Nico and Katya to determine if baking these metrics directly into the codebase would be more efficient than the current log-based approach[1].
ChainSafe infrastructure presented a proposal to implement OpkSSH (similar to OIDC) for server authentication, which would provide temporary keys with periodic re-authentication (configurable intervals from 1 day to 1 week). The main benefits would be improved security posture and simplified management when team members leave. However, the proposal met significant resistance from the development team:
Matt and Cayman strongly opposed the change, arguing that:
- The team is small (7-8 people) with low churn (only 1-2 people leaving per year)
- Current workflow efficiency would be compromised
- Security risk on development servers is minimal (no economic value)
- Production servers already have separate, tighter security patterns
- The added friction outweighs the benefits for our use case
The infrastructure team ultimately accepted the feedback and withdrew the proposal.
Phil highlighted the significant interest generated by Lodestar's announcement of going hybrid ZIG/JavaScript, evidenced by new contributors joining the Discord. The main catalyst goal is running Lodestar on Bun runtime, with a comprehensive checklist created to supersede previous V2 targets.
Cayman presented a visual diagram showing the ZIG transformation architecture, illustrating:
- ZIG libraries that will be native
- ZIG libraries exposed to Lodestar via FFI
- JavaScript layer components
- Two parallel work streams: component integration and rearchitecting (with rearchitecting dependent on integration progress)[1]
A significant discussion centered on the choice between Bun FFI and NAPI for the binding layer:
Arguments for Bun FFI:
- Performance benefits (2-6x faster than NAPI in benchmarks)
- Lower overhead for frequent operations
- Direct pointer manipulation capabilities[1]
Arguments for NAPI:
-
Nazar shared concerning experiences with Bun, including:
- Fundamental issues with module loading (top-level await problems)
- 4,600+ open issues on Bun's repository
- Difficulty getting critical issues resolved
- Unstable runtime behavior[1]
- Phil noted warnings from other developers (Fucory of TEVM project) about Bun's long resolution times for issues
- Runtime compatibility concerns with maintaining both Node.js and Bun support
Team Consensus: The team agreed to proceed with NAPI for the binding layer to maintain compatibility with both Node.js and Bun runtimes, with the option to add specific Bun FFI bindings for performance-critical operations if needed.
Cayman provided updates on ZIG development:
- ZIG 0.15 targeted for August 1st release
- Async Await revival discussed in Andrew Kelly's recent live stream, following an allocator-style pattern where IO objects are passed into functions and then you allocate with it. Likewise, if you want to do IO, you're going to pass in an IO object into your function and then do IO with it.
- This will address a missing piece for building large products in ZIG
Cayman reported on research into the Constantine BLS library by ex-Nimbus developer Mamy:
- Features comprehensive thread pool implementation with detailed documentation
- Pre-1.0 status due to pending security audit
- Active development with audit in progress
- Team will monitor for post-audit availability
- Cannot be used until security audit is completed
NC provided an update on the proposer boost feature testing:
- Deployed to feat4 validator for approximately 6 days on mainnet
- Caught 6-7 weak/late blocks and successfully suppressed FCU calls
- Challenge: All captured blocks were too late (13+ seconds, extending into the next slot), making it impossible to test the actual reorg functionality
- Some "unknown" reasons for FCU suppression logged, requiring further investigation
- Phil removed the feature from the 1.32 milestone due to testing difficulties
The team discussed the 1.32 release status:
- 36 commits ready for release
- Key features include:
- KZG switch to Rust implementation
- Default gas limit increase to 45 million
- Proposer boost feature removed from milestone due to testing challenges
- Tuyen emphasized prioritizing PeerDAS refactor PRs merging to unstable after release
- Matthew Keil noted several refactor PRs ready for review
Matthew mentioned merging work from Fusaka DevNet 2 into unstable branch, including:
- BPO fork functionality
- Fork digest updates
- Consider what else needs to land on unstable and move it over to unstable
- NC inquired about package structure for the new ERA format for blobs, with Cayman recommending consolidation of all E2Store/ERA functionality into a single package.
- We want to support the format and help where possible on speccing ERB for blobs: https://github.com/status-im/nimbus-eth2/pull/5882
- The team discussed Ethereum execution clients' implementation of history pruning for pre-merge data on mainnet, with external history providers now becoming necessary for archive node users and application developers.
- History providers can be found here: https://eth-clients.github.io/history-endpoints/
Nazar provided updates on Bun-related development:
- Unit test PR for Bun runtime ready and waiting for review
- Benchmark comparison PR in development to compare Node.js vs Bun performance, starting with state transition benchmarks
- Emphasis on getting concrete performance numbers rather than assumptions[1]
Phil celebrated Lodestar being the first client team to achieve all green status on all viable test combinations in fusaka-devnet-2
Agenda: https://github.com/ChainSafe/lodestar/discussions/7885
Recording/Transcript: https://drive.google.com/drive/folders/1ydcYmPXusmjj8PAQyqd2V-nyPqcwfEfG?usp=sharing
The team discussed scheduling for the upcoming weeks due to ZuBerlin attendance.
The next standup was tentatively scheduled for July 8th, with the possibility of async updates during the interim period.
- The team finalized preparations for release 1.31, with Nico confirming the final PR had been merged. The release strategy involves cutting from unstable, creating an RC1 for sanity testing, then shipping.
- The release primarily contains UX improvements with minimal functional changes that would affect core operations.
- Post-release, the team plans to deploy to Lido to gather additional metrics and block data.
- Based on conversations with Toni, the team discussed removing min-bid requirements on their Lido validators and connecting to the top four relays to reduce latency.
- They evaluated different relay options, noting that Titan offers both filtering and non-filtering relays, with preference for non-censoring variants.
- For Bloxroute, they identified avoiding the regulated relay in favor of their max profit option.
Assignment to sessions were noted and members were assigned accordingly to ensure we have coverage on all important breakout sessions. Please see internal chat for scheduling.
Cayman emphasized the importance of preparation, noting that attendees should read associated documentation before sessions to contribute meaningfully rather than passively absorbing information[1].
- The team discussed significant changes at the Ethereum Foundation, including the elimination of several teams and reorganization into three initiatives: Scaling L1, Scaling L2/Blobs, and UX.
- Notable eliminations included EthereumJS (despite millions of monthly downloads) and the entire Portal Network team.
- The team expressed surprise at these decisions, particularly given EthereumJS's usage and their previous assumptions about Portal Network integration with history expiry.
- This reorganization appears to shift EF focus more toward research rather than implementation and tooling.
- Nico identified several items needed for DevNet preparation, including merging unstable into PeerDAS branch to include SSZ updates, digest mismatch fixes, and blob schedule improvements to maintain spec compliance.
- Matthew committed to handling the merge and addressing identified conflicts.
- The team also discussed the need to merge proposer look-ahead functionality into unstable and subsequently port it to PeerDAS for fusaka-devnet-1.
Matthew provided extensive analysis of validator custody implementation challenges.
- While the team has working validator custody functionality, they face significant challenges with the 18-day expiry window for DevNet participation.
- The primary blocker is the lack of robust backfill capabilities, which Matthew described as a "fairly substantial body of work" that is "going to be pretty bugprone".
- The current approach will involve enabling validator custody (available as a simple flag) but not announcing custody changes to avoid penalties during the initial period when historical data isn't fully available.
- Matthew noted that the existing backfill code needs substantial refactoring, recommending a clean-slate approach using current range sync methodologies rather than the complex mix of range sync and root sync currently implemented.
- Phil raised the topic of the proposed 60 million gas limit increase, noting discussions from the ACDT call about EF signaling support.
- The team confirmed this falls within CL gossip limits (which cap around 65-70 million gas before hitting 10MB block limits).
- They plan to monitor existing HoleΕ‘ky or Sepolia nodes for performance data before implementing any default changes.
- The team scheduled a dedicated roadmap discussion for Monday evening (17:00-18:30) during the Berlin interop period.
- This session will focus on technical direction decisions including the transition to Bun runtime and Zig-based library development, incorporating findings from Cayman, Tuyen, and Nazar's research into Lodestar's future architecture.
Matthew:
- Matthew reported significant progress on by-range functionality debugging, including identification and resolution of multiple bugs through unit testing.
- He encountered a challenging out-of-memory issue that consumed two days of debugging time, ultimately traced to fixture building processes.
- Despite the debugging challenges, unit testing is proceeding smoothly and the codebase appears stable.
- Matthew confirmed completion of the merge from unstable and readiness to proceed with additional small PRs.
Tuyen
- Tuyen reported progress on data column site guide implementation and upcoming work on subnet subscription optimization.
- He's working on preventing nodes from subscribing to their own subnets, with an existing PR available but blocked by current data column work.
- The optimization focuses on topic-based subnet maintenance to improve network efficiency.
Nico
- Nico provided updates on specification compliance gaps, identifying several areas needing attention beyond backfill capabilities. = He highlighted missing attestations as potentially more critical than backfill issues.
- Additionally, he identified a timing issue in block building where header requests are delayed after block body construction, which could be interpreted as timing games and violates builder specifications.
- Nico emphasized that while backfill is the major missing component, these other specification gaps also require attention.
NC (Async)
- Reviewed the EIP-7917 spec. Should be ready to merge today,
- EIP-7917 PR is up https://github.com/ChainSafe/lodestar/pull/7902 . Already implemented it, just need to spec test it
- Barnabus is trying to test fusaka devnet-1 which is pretty much 7917+peerdas.
- A temporary branch is set up for him to test https://github.com/ChainSafe/lodestar/pull/7904,
- Still need to address the proposer boost reorg fix PR.,
- Still need to review diff archive repo PR,
- Going to find some time later this week to update the fork digest for BPO (this is for devnet-2)
Agenda: https://github.com/ChainSafe/lodestar/discussions/7851
Recording/Transcript: https://drive.google.com/drive/folders/17k_ReeYL9JjGBZ84t7eRimaMHaT9Y3ht?usp=sharing
Phil initiated discussions about coordinating an Ethereum guild meeting with Toni from the Ethereum Foundation to discuss Lodestar's Lido validators and their impact on the network.
- Meeting is tentatively scheduled for June 3rd
- The team also discussed the Ethereum Season of Internships program, with Phil clarifying that they can host interns and EPF fellows as long as they work on different projects, contingent on team bandwidth and establishing minimum qualifications for engineering proficiency.
- The team evaluated implementing a CI workflow for nightly spec test vectors, which would track unstable test vectors that may frequently fail. -Despite concerns about notification management and frequent failures, the consensus was to proceed with the implementation if the team finds it useful for development purposes.
- A significant discussion focused on sporadic attestation performance issues identified by Sam at ETH Panda Ops following the Pectra upgrade.
- Analysis of validator indexes revealed that Lodestar nodes are experiencing inconsistent attestation inclusions, with some blocks showing only 40% participation in attestation zero while sync aggregates maintain 99% participation.
- This represents a critical mainnet performance issue that appears sporadic rather than systematic.
- Nico confirmed that individual validator performance remains excellent, with his validators achieving top 1% effectiveness ratings.
- The problem specifically relates to attestation packing during block production, where attestations from the previous slot are being omitted inconsistently.
- The team identified the need to implement comprehensive metrics dashboards to analyze:
- block production timing
- parent block processing
- potential correlation with network conditions
- reorgs
- Matthew and Tuyen provided detailed analysis of performance bottlenecks identified through Sunnyside Labs testing.
- Key optimization opportunities include:
- batching KZG commitments, which could save approximately 160 milliseconds per operation
- implementing signature verification caching to avoid redundant BLS verifications for identical block headers and signatures.
- The team discussed CPU usage concerns, particularly related to subscribing to all 128 subnets instead of the standard 8 for normal nodes. - Tuyen identified message ID inconsistencies in gossip sub implementation that may be contributing to low peer scores and reduced mesh participation.
- The team acknowledged that while
fusaka-devnet-0is functioning, performance optimizations are critical for mainnet readiness.
- Work allocation for Berlin interop was discussed, with three main focus areas identified:
- performance tuning and networking issue resolution,
- PeerDAS implementation completion and testing
- comprehensive unit testing of the new block input system.
- The team emphasized the need for better preparation compared to previous Electra work
-
A critical discussion addressed the AVX instruction crash affecting users without AVX support in their processors.
-
Matthew presented two potential solutions:
- a short-term fix using CPU detection in the application layer with the existing
nanlibrary (which breaks Bun and Deno compatibility), - a longer-term solution implementing CPU detection in the Rust binding layer using napi-rs.
- a short-term fix using CPU detection in the application layer with the existing
-
Cayman advocated for the Rust-based solution to maintain cross-runtime compatibility, while Matthew emphasized the urgency of addressing user crashes before the Berlin interop.
-
The team reached consensus to implement Matthew's temporary fix for the upcoming release candidate hotfix, with plans to develop the proper Rust-based solution post-interop.
-
This decision prioritizes immediate user experience while acknowledging technical debt.
- With team members traveling to Berlin and potential participation in subsequent events, the window for release management becomes constrained.
- The consensus was to cut a hotfix 1.30.1 release candidate addressing the AVX crash issue, as this represents a critical user-facing problem that has caused users to switch to alternative clients.
- Phil emphasized that without specific planning, the next release opportunity may not occur until after the Berlin interop period, making the current release candidate crucial for addressing known issues before the team's availability becomes limited.
Matthew
- Matthew reported significant progress on cache PR implementation, incorporating feedback from Cayman and Tuyen with comprehensive unit testing additions.
- He completed unit testing for the block input system, discovering and fixing minor bugs in the process.
- Range code unit testing is substantially complete with only minor remaining work.
- His investigation into the AVX crash issue in SSZ resulted in a working CPU detection solution, though it introduces compatibility issues with Bun and Deno runtimes.
- He researched native C library alternatives and identified suitable options for long-term implementation.
- Matthew emphasized the need to balance immediate user needs against technical debt considerations for the upcoming release.
Tuyen
- Tuyen focused extensively on networking investigation for PeerDAS, discovering message ID inconsistencies in gossip sub implementation when connecting to different client types.
- His analysis revealed that the publish function in gossip sub was not updating the message tracer, for which he submitted a PR.
- He also identified duplicate message sending to peers and opened a PR to address sync contribution and proof metrics.
- His attempt to merge v1.30.0 into the Bun branch encountered significant issues with high CPU usage and frequent disconnections, despite previous successful testing.
- Given the focus required for PeerDAS optimization before interop, he decided to deprioritize Bun development in favor of mainnet performance issues.
Nico
- Nico completed work on blob schedule functionality and addressed debugging issues raised by Barnabas, determining that reported problems were primarily on the external side rather than Lodestar implementation.
- He identified an orphan block issue on the PeerDAS branch where block imports were occurring 11-12 seconds into the next slot, suggesting potential problems with the unknown block handling or peer discovery mechanisms.
- His priorities for the current week include investigating the attestation packing issue and potentially implementing his minimal approach for parallel block processing refactoring.
- However, he emphasized that bug fixes must take precedence over major refactoring efforts until critical issues are resolved.
NC
- NC completed the proposal boost rearchitecture PR and addressed review feedback from Nico.
- He opened an issue with the Attestant go-eth2 client regarding blob schedule format support in the beacon API, which affects the nightly Kurtosis CI testing.
- This external dependency issue may impact other teams using similar testing infrastructure.
- With available capacity for the current week, NC offered to contribute to either PeerDAS development or attestation-related issue investigation.
Cayman
- Cayman achieved an additional 40% performance improvement in SSZ tree navigation
- Collaborated with Matthew on block input unit testing, specifically focusing on get block by route functionality.
- He worked with external teams on SSZ EIP development, planning collaboration opportunities for the Berlin interop period that may result in stable container EIP modifications.
- His continued focus remains on block input system development and providing support where needed for the team's interop preparation efforts.
Katya
- Katya opened a PR implementing getBlobv2 metrics to enable comparison between gossip and direct blob retrieval performance.
- She continued development of validator activity metrics with significant refactoring collaboration with Nico.
- Her work on the Grafana dashboard script addressed testing issues that were subsequently merged.
- Volunteered to assist with performance investigation despite limited experience with specific metrics and monitoring, demonstrating willingness to learn debugging processes through shadowing experienced team members.
Gajinder
- Gajinder worked on optimizations for data column sidecar requests but encountered resistance from other client teams who prefer batching blocks from different DAS sources via gossip rather than maintaining column consistency.
- The optimization PR may be dropped pending broader client team consensus on the approach.
Agenda: https://github.com/ChainSafe/lodestar/discussions/7833
Recording/Transcript: https://drive.google.com/drive/folders/11K-YGskKUk-V_wzX0HT-jst6eII5eaLz?usp=sharing
Fusaka Progress & Berlin Interop Readiness
- The team is focused on completing Fusaka-related goals ahead of the Berlin interop event.
- Key goals include:
- Completing the refactoring of the DA (Data Availability) module.
- Merging the
peerDASbranch intounstable. Progress is ongoing; most PR comments have been addressed, with unit testing and final reviews underway. Some unit tests are being added to thebyRootandbyRangeroutes. Cayman is assisting with reviews. - Performance degradation has been observed in the
peerDASbranch, but this will be addressed after functional code is merged and stabilized. - There is discussion on whether merging to
unstablebefore interop is necessary. The main concern is spec test downloads, but recent improvements mean download speed is no longer a bottleneck. The consensus is to ensure the code is in a merge-ready state, even if not merged, to minimize risk during Berlin. - NC will reuse the
peerDASbranch for Fusaka-Devnet-0, and we will cherry-pick commits such as BPO fork as necessary.
DevNet and Gossip Score Issues
- Tuyen noted low gossip scores on DevNet, particularly for p7 where there are a lot of broken
IWANTpromises due to slow responsiveness and not related to main thread. - There are ongoing efforts to update to the latest consensus spec tests, though some new test runners and processing logic require more sophisticated updates to pass.
EF Security "Trillion Dollar Initiative" Feedback
- The EF Security team is soliciting feedback from client teams on improving Ethereum security, including user experience (UX) concerns.
- Team members highlighted persistent wallet UX issues:
- Lack of transparency in what users are signing, particularly with complex contract interactions and account abstraction.
- Even advanced hardware wallets often resort to blind signing for contracts, which is a security risk.
- The idea of a "debug transaction" RPC endpoint was discussed, allowing users or wallets to simulate transactions and see their effects before signing. Existing tools and UIs (like MetaMask) already simulate transactions, but trust in the UI remains a concern.
- The team acknowledged that while simulation tools exist, reliance on UI can still be a vector for attacks (e.g., Safe wallet incident), and more robust, protocol-level solutions may be needed.
Lido Min-Bid and APR Discussion
- The team revisited the use of the
min-bidparameter in Lido node operations, noting that changing network conditions (e.g., more local blocks, lower MEV) have made the current value potentially outdated. - It was observed that the team's APR is about 0.3% lower than top operators, despite similar effectiveness, likely due to the min-bid setting.
- There is interest in collecting more data (e.g., from rated.network and Lido's reporting) to inform whether the min-bid should be adjusted for better network alignment and operator competitiveness.
- The difference between builder and local block values has increased, raising questions about the current selection logic and whether it is still optimal.
P2P Config/Fork Handling and BPO Forks
- Ongoing discussions about how to handle configuration mismatches and fork awareness in the P2P protocol, especially as BPO (blob per block parameterization) and future forks proliferate.
- Gajinder proposed an EIP to canonicalize and Merkle-ize the active configuration, allowing handshake-based config verification between peers, reducing the risk of accidental peering across incompatible forks.
- There is some skepticism about feasibility for the Fulu fork, but consensus that long-term, protocol-level config verification is needed for network health and security.
Testnet Gas Limit Increases
- Phil checked with the team about increasing gas limits on the Hoodi testnet to 60M, noting no issues so far and alignment with Sepolia's current settings.
Sync Aggregate Issue
- Nico raised a concern about mainnet blocks with zero sync aggregates, an unexpected situation possibly introduced by recent changes (e.g., Pectra). The team agreed to open a thread for further investigation.
Cayman
- Completed major work on the SSZ-zig library's persistent Merkle tree, achieving 5β15x performance improvements in tree navigation.
- Ready to merge a large refactor PR; will shift focus to block input refactor to help accelerate progress.
- Released
bun-ffiutility library v1.0, enabling faster and more efficient use of native modules with Bun.
Tuyen
- Worked on Bun bindings with Cayman; only upgrade Bun FFI to the correct release version remains before merging and releasing.
- Preparing to integrate into the
peerDASbranch; merged unstable into peerDAS, with the PR ready and green. - Fixed an issue with PeerDAS long job wait time due to
isProcessingCurrentSlotBlock. Context can be found here: https://github.com/ChainSafe/lodestar/issues/7847#issuecomment-2893179748 - Gossip score in PeerDAS is low, that's why we're not acccepting messages from some peers, should be resolved before the interop.
- We don't run the merkle spec test in peerDAS. Discovered when trying to merge unstable in to peerDAS.
- When we get the merkle proof, we fail it - might be an issue in SSZ but not sure why because it's been stable for a while.
- Missing new
networkingspec tests in peerDAS: https://github.com/ChainSafe/lodestar/issues/7839
Matthew
- Progressed on Fusaka PRs, with unit tests and reviews ongoing.
- Accidentally squash-merged a branch intended for regular merge; will revert and re-merge as required.
- Continues to target merging into unstable before Berlin.
Nico
- Reviewed BarnabΓ‘s's blob schedule work; branch is ready pending review and interop considerations with other clients.
- Plans to resume work on block parallelization after DevNet zero is ready.
- Discussed the need for improved fork/config handling in P2P.
NC
- Worked on BPO PR, tested with BarnabΓ‘s's config, and followed BPO P2P discussions.
- Updated Deneb attestation code and error messages, merged related PRs.
- Fixed rewards API pre-state calculation.
- Kurtosis CI test PR merged and running smoothly.
- Reorg proposal PR will now include both refactor and fix, as little refactoring was needed.
Gajinder
- Opened a PR for consensus spec optimization, specifically around data sidecar byRoot request/response.
- Provided feedback on Justin's KZG crypto crate.
- Participated in BPO fork discussions, advocating for handshake-based config verification.
- May draft an EIP to formalize Merkle-ized config for peer handshakes with Nico.
Katya
- Completed validator monitor refactor; follow-up PRs are open and approved.
- Set up Ansible and resolved Python version/server storage issues.
- Cleaned up server images and restored validator operations.
- Working on pure dust metrics and a Kurtosis dashboard; validator activity metrics PR is ready for review.
Nazar
- Investigated Bun vs. NodeJS runtime discrepancies (e.g., MessageChannel differences) impacting tooling.
- Developing polyfills to bridge compatibility gaps.
- JSR publishing PR for SSZ packages pending Cayman's review; differential states PR reviewed by Matthew, awaiting further feedback.
Agenda: https://github.com/ChainSafe/lodestar/discussions/7807
Recording/Transcript: https://drive.google.com/drive/folders/1YF_GCA4aFp037AIr_aYUax4Ljtc0LoMD?usp=sharing
- Matthew reported that PeerDAS DevNet7 is running smoothly with all monitored nodes performing well along with moving keys over to our local validators
- Cayman's PR for PeerDAS has been merged, with Matthew working on a follow-up PR that will be pushed shortly
- Team expects PeerDAS integration into the unstable branch within approximately two weeks
- Integration work is being done incrementally with most components being merged directly to unstable as the refactoring progresses
- The team performed an initial review of version 1.30 with no major issues detected
- Nico identified one broken VM host panel issue, but this is unrelated to core functionality and will be fixed separately
- Team noted that post-Pectra performance metrics show Lodestar performing better than before, while the network average has actually declined
- Performance improvement appears to be related to optimized attestation handling, placing Lodestar in a favorable position compared to other clients
- Chiemerie is setting up Blockprint for both the Execution Layer (EL) and Consensus Layer (CL)
- The team discussed using the Blockprint's Blockdreamer for triggering block production, which Nico confirmed had been recently updated (within 10 hours) to add Electra support
- Plan is to extract and trigger the build process to send blocks to the SigP Blockgauge instance rather than setting up a separate block print instance
- Team decided to implement a prudent strategy regarding Electric Eel (Eleel) integration: start without it and implement only if block production degradation is observed
- Concerns were raised about potential performance with blocks every 12 seconds, which may require Eleel faster block creation capabilities
- Nazar and Nico presented different approaches to parallelizing block production steps
- Two competing strategies are being considered: a minimal approach versus more extensive refactoring
- Nazar believes refactoring is needed due to the code's complexity
- Nico is concerned about introducing regressions.
- Matthew also agreed that refactoring is a good idea but echoed concerns about the complexity and suggested a long testing period for any changes
- Team acknowledged that the current block production code path is complex with up to 6-8 levels of indentation, making it difficult to read, understand, and debug
- Matthew suggested that any refactoring should be thoroughly tested over an extended period (up to a month) to ensure no regressions occur under various network conditions
- Cayman proposed using Blockdreamer to test block production every slot for comprehensive validation
- Discussion will continue asynchronously in a dedicated Discord thread to determine the optimal approach
- Team discussed how the proposed booster should work according to specification requirements
- Technical conversation centered around ForkchoiceUpdate (FCU) call timing and suppression strategies
- NC and Nico agreed on when to issue and suppress FCU calls, particularly regarding previous slot and T0 behaviors
- Team considered Lighthouse's approach of issuing another FCU call 500 milliseconds before the slot
- The discussion continued around ensuring the EL (Execution Layer) does not reorg due to multiple FCU calls.
- Tuyen suggested issuing two FCU calls with different payload IDs, but NC believed the EL would ignore the second call
- Nico clarified that the EL can be told to prepare a payload for the parent and later for the weak block without causing a reorg
- NC opened a draft PR implementing Kurtosis CI nightly tests, which currently run in approximately 18 minutes
- Additional test suites from external sources are available but not yet integrated, including Pectra, maxEB, and consolidation tests
- Team decided to schedule the automated CI action at midnight UTC to minimize disruption to development workflows
- Infrastructure will be adjusted if timing becomes problematic for team members across different time zones
- Chiemerie acknowledged Cayman's request about sharing dashboards publicly
- Work is underway to provide public access to Grafana dashboards before the week ends
- Team discussed the feature availability on both self-hosted and cloud instance
Matthew
- Successfully maintaining PeerDAS DevNet7 with all monitoring showing positive results
- Rebased and fixed issues in a PR that will be pushed within hours
- Preparing additional PR with components already broken out from previous work
Nico
- Investigated approaches for block refactoring parallelization, focusing on understanding constraints before proceeding with implementation
- Reviewed the proposer boost topic specification and implementation details
- Noticed that Lodestar's performance improved post-Pectra while the overall network performance declined
- Observed issues with Geth during reorgs when trying to incorporate multiple FCU calls
- Working on various PR reviews
Nazar
- Working on integrating Bun-related unit tests into the Lodestar repository to track working and broken components
- Developing a PR related to state diffing that will be opened soon
- Advocating for refactoring complex block production code paths to reduce technical debt
- Created simplified code snippets to help team members evaluate different parallelization approaches
NC
- Working on proposer boost reorg architecture implementation details
- Set up Kurtosis CI integration with basic tests completing in approximately 18 minutes
- Planning to add additional test suites for Pectra, max EB, and consolidation from available resources
- Reached consensus with Nico on FCU call timing strategies for proposed booster implementation
- Preparing to configure the CI workflow to run at midnight UTC
Tuyen
- Completed work on building for State Transition util with Bun integration
- Developed a tool to handle publishing Bun libraries for different platforms
- Working on implementing top-level await functionality based on Cayman's feedback
- Migrated BLST-Z version to Zig 0.14
- Fixed merge issue for PeerDAS
- Identified performance issue with validating messages in PeerDAS taking approximately 30 seconds to execute
Katya
- Set up mainnet beacon node for testing and monitoring
- Facing configuration challenges with Grafana when attempting to run both Hoodi and mainnet nodes simultaneously
- Completed validator monitor metrics reorg
- Discussing options with Nico for either adding network labels to Prometheus config or running separate Prometheus/Grafana instances
- Planning to use Ansible for validator metrics configuration
Cayman
- Successfully merged core data structures for the data availability and block input refactors with Matthew's help
- Creating tooling to support Zig projects with platform-specific library builds similar to the NAPI-RS approach
- Starting work on persistent Merkle tree implementation for SSZ-zig
- Developing proof infrastructure to differentiate Lodestar from other clients
- Supporting Tuyen with Bun integration work
- Highlighted Bun's advantages including built-in S3 client capabilities and better tooling
Chiemerie
- Setting up both Execution Layer and Consensus Layer for Blockprint
- Researching options for Electric Eel implementation
- Working on providing public access to shared Grafana dashboards
Mercy (External Contributor)
- Working on optimizing node traversal by combining get-all-nodes functionality into a single traversal method
- Seeking guidance on implementation approach based on team feedback
The meeting concluded with a final note to have Tuyen and Cayman review metrics for the upcoming release, and Tuyen highlighting a significant performance issue with message validation in PeerDAS that requires attention.
- Matthew to complete and push PeerDAS-related PRs
- Team to review 13.0 release metrics before final approval
- Chiemerie to implement public access to Grafana dashboards
- Further discussion needed on block production parallelization approach
- Matthew to investigate performance issue with PeerDAS message validation
- Cayman to provide reviews for Matthew's upcoming PRs when ready
Agenda: https://github.com/ChainSafe/lodestar/discussions/7778
Recording/Transcript: https://drive.google.com/drive/folders/1dW0nBmlAO_Wt_gBemSnQbajgtvhhUrjQ?usp=sharing
- Team confirmed all Pectra-related preparations are complete
- Pectra fork is scheduled to occur in approximately 20 hours
- Phil acknowledged all mainnet preparations have been handled by the infrastructure team
- Team agreed to target a release before merging PeerDAS changes
- Aiming to cut RC on May 9th (Friday) with a potential v1.30 release on May 13th
- Cayman emphasized the importance of maintaining the regular release cadence
- Proposer boost feature was deprioritized over concerns about its implementation:
- Nico noted it's a significant change despite small PR size as it "triggers a lot of code"
- The implementation might not be correct based on current reviews
- Discussion on potentially including async aggregate and snappy changes:
- Nico shared metrics from Gnosis nodes showing promising improvements
- Metrics show improved event loop lag performance, especially important for 5-second slot times
- Team will review additional metrics before deciding
- Discussed integration of Kurtosis tests alongside current sim tests
- Team agreed to maintain current sim tests as lightweight sanity checks for PR validation
- Will implement more comprehensive Kurtosis tests to run nightly rather than on every PR
- Focus on basic functionality rather than extensive assertions to prevent test instability
- NC shared examples from Reth team's approach of running tests every 12 hours
- How much do we want to assert for every PR and nightly?
- A simple sanity check: Can the chain progress? It should at least progress and finalize.
- Maybe consider the top 3 assertions and use those?
- One request that was made from NC was for sim test assertions with execution requests (calling, sending and checking deposit transactions)
- There is some value in the current sim tests though, like how we found out Lighthouse deprecated mplex.
- Sim tests: Should be interop related tests. Can we peer with Lighthouse? Can we work with mixed clients? Can we interop with Geth/Nethermind?
- Kurtosis tests: Nightly cron job run with more assertions.
- We may have self-hosted runners we should look into specifically for purposes like this.
- Nazar mentioned fixing e2e leaky tests (PRs #7762 and #7780)
- Team confirmed agreement to extract simple-serialize.com from the main repository
- Nazar received confirmation to preserve git history during the extraction process
- Plan is to add an extra remote to the new repo and push the master branch
- Team acknowledged the website needs improvement, but extraction will increase exposure to front-end developers
- Discussion about whether to use minimal or mainnet preset for unit tests
- Nico investigated and found minimal performance difference between presets for unit tests
- Team agreed to use mainnet preset for unit tests as it's easier to reason about (32 slots vs. 8 slots)
- Will continue using minimal preset for sim/e2e tests where the performance improvement is significant
- Some tests that are more end-to-end in nature will be refactored and moved accordingly
- Katya requested feedback on WiEP contributors
- Nico provided feedback that contributors are enthusiastic but face technical knowledge barriers
- Suggestions for improvement:
- Prepare easier "good first issues" that are additive rather than refactoring
- Create fresh issues rather than using old ones
- Potentially create onboarding materials specific to Lodestar structure
- Consider recordings explaining the repository structure
Matthew PeerDAS & DevNet Updates
- Made progress on block input refactoring PRs
- Blob code changes are being merged into unstable branch
- Work on range and gossip syncing mechanisms progressing
- Performance testing planned for next week once changes are merged
- DevNet 7 being planned for this week after Pectra fork
- Sunnyside network crashed and stopped syncing, team working on identifying the issue
- Bug likely related to validator custody changes and column retrieval from database
Tuyen State Transition in Zig
- Made progress on state transition utilities in Zig
- Identified performance issues with SHA256 implementation:
- Shuffling is 20% slower compared to Rust implementation
- Committee indices computation is 40% slower
- Fixed build issues in the hash tree implementation
- Added unit tests and benchmarks
- Expects to release the beta version within the week
- Plans to start integration with Bun branch next week
Derek PeerDAS Updates
- Available to help with DevNet 7 branch
- Temporarily reassigned to work on general testing of the network
- Will focus on improving stability and Kurtosis tooling
- Working on identifying syncing issues across clients
- May help with ProbeLabs testing to verify clients are properly custodying advertised data
Katya Validator & Beacon Node Metrics
- Updated beacon node for Petra using quickstart
- Working on moving validator monitor metrics
- Planning to implement Tuyen's validator metrics after current PRs are merged
- Received guidance on what to monitor during Petra fork (blocks, attestations, justification, finalization)
Gajinder getBlobsv2 Metrics
- Shared metrics discussion from recent call
- Reported 60-70% hit rate for getBlobs v2, validating the team's observations
- Analysis showed that in 30% of failed cases, typically one blob was missing
- Block blob propagation metrics show improvement:
- Previous: 700ms for six blobs at 95th percentile
- Current: approximately 200ms with aggressive pull strategies
- Working on spec PR to improve get data column sidecar API
- Engaged in discussions about edge cases for dynamic validator custody updates
NC Proposal Boost Rework
- Reviewing proposal boost rework implementation
- Nico noted importance of first determining theoretical FCU call patterns before implementation
- Attended FOCIL breakout session which presented a solution for delay execution coexisting with FOCIL
- This could potentially allow both EIPs to go into Glamsterdam
Chiemierie
- Chiemerie confirmed he'll look into Kurtosis setup
Glenn
- Glenn introduced himself as taking over communications
- Mentioned upcoming community update next week and plans for internal newsletters
Cayman
- Working on data availability refactoring and Zig build tools
Nazar
- Merged two PRs related to binary improvements, with two more coming this week
Nico
- Nico: Merged various PRs and added tests for "staring" functionality
Agenda: https://github.com/ChainSafe/lodestar/discussions/7739
Recording/Transcript: https://drive.google.com/drive/folders/1YYXA-M-7aeaZeFxLvmju2EGTsRZ_axjN?usp=sharing
- Lodestar team is "in the middle of the pack" with PeerDAS implementation and ready for the next devnet-7.
- All necessary PRs have been merged, with a few more open from Derek and Hugh.
- The team has been conducting extensive testing with Kurtosis to ensure everything works smoothly.
- Standby for another list of remaining items from Derek relating to PeerDAS work. Will be available on Discord as needed but ramping down contributions.
A significant technical challenge discussed was column syncing, which Derek described as deceptively complex despite being just "a sentence in the spec." The team identified several implementation challenges:
- Managing peer prioritization when syncing columns
- Handling scenarios where specific columns aren't available from peers
- Decision points around whether to block syncing or allow blocks to run ahead without data columns
The team also discussed issues observed in DevNet-6:
- Validator custody operations ran "wild" with fluctuating custody group counts.
- When nodes fell behind, they couldn't catch up because "nobody knew who had what columns anymore".
The team discussed two major goals to accomplish before the Berlin interop in June:
- Complete the refactoring of DA (Data Availability) modeling to make it more conceptually clear and easier to work with
- Get the PeerDAS branch merged into unstable
- Matthew reported progress on the first piece of the refactor code structure, having gone through three different versions before settling on the right approach. He's currently working on block synchronization components including "by range," "by route," and "unknown block input sync".
- Phil emphasized the importance of improving visibility into their own codebase to better identify problems during the interop. The team agreed that having PeerDAS merged into unstable before the interop would significantly improve their ability to iterate quickly without having to switch between branches.
- We should also target unstable for further Fulu related implementations as soon as we can get PeerDAS branch merged into unstable (after mainnet Pectra fork and stability). Cayman noted that while the timeline is tight, both goals are achievable and would greatly benefit their workflow during the interop.
- The team discussed setting up Gnosis nodes to gain performance insights, particularly around handling shorter slot times.
- Nico emphasized that running Gnosis nodes in their test infrastructure would provide valuable data on how different slot timing affects Lodestar implementation
The team discussed potential approaches:
- Running a basic node without subscribe all subnets
- Potentially running validators (requiring GNO tokens at approximately 130 USD per validator)
- Reaching out to the Gnosis Foundation to potentially delegate validators without requiring token ownership
- We will run a
gnosisgroup of servers that will collect data on our performance on the Gnosis network with and without subscribe all subnets.
Nazar reported investigating flaky end-to-end tests that have been causing issues. The team debated whether to fix, delete, or modify these tests:
- Nazar noted that while flaky, the tests cover important edge cases that verify specific error conditions crucial to network request flow
- Nico observed that the tests appear to fail due to timing/race conditions, particularly when CI runners are under load
Several potential solutions were discussed:
- Moving expensive/flaky tests to nightly cron jobs
- Using beefier CI runners
- Implementing self-hosted runners to eliminate GitHub runner-specific issues
- Identifying specific configuration mismatches causing connection refusal errors
Matt suggested the possibility that some issues might be related to how GitHub builds runners, potentially causing cross-VM interference.
Nazar will continue investigating and we can conduct experiments with different hardware if we feel it is a cause of the issue.
Mercy (external contributor) brought up work on validating Genesis state roots across different networks. The team discussed the value of embedding state roots in the client:
- Nico explained that currently we "yolo download the state from GitHub" without validation, and having Genesis state routes baked into the client would improve verification
- The team debated extending this approach to weak subjectivity checkpoints, with g11tech suggesting that regular releases could include updated checkpoints as a value-add
- Concerns were raised about baking in checkpoints potentially undermining the community consensus aspect of weak subjectivity, with Nico noting "the point of weak subjectivity is that it should be rough community consensus around the right chain, not controlled by five teams"
Cayman
- Completed work on the Zig SSZ implementation, passing all spec tests with promising benchmark results
- Ready to merge the refactored branch to the main branch
- Next step is creating codegen to expose the library to Bun via a NAPI-compatible interface
- Made progress on libP2P-quic implementation but encountered segfaults requiring further investigation
Nazar
- Worked on parallelizing block processing and production to improve performance
- Implemented processing steps for generating common block body and builder/engine block body in parallel
- Received and addressed feedback from Tuyen on the implementation
- Currently investigating flaky tests focusing on connection issues in the libp2p layer
Nico
- Completed cleanup work after the Electra release
- Removed produce block v2 APIs to simplify the codebase
- Investigated performance issues reported by Luca on Gnosis, identified high CPU usage in "aggregate with randomness" operations
- Reviewing and merging various PRs
Katya
- State transition implementation merged with a follow-up issue identified for later work
- Currently working on validator monitor metrics implementation
- Approved for attendance at protocol.berlin
Derek & Hugh (Base Contributors)
- Continuing testing of PeerDAS implementation this week
- Will create comprehensive documentation of remaining work needed for PeerDAS
- Transitioning off the Lodestar project but will remain available via Discord
Agenda: https://github.com/ChainSafe/lodestar/discussions/7706
Recording/Transcript: https://drive.google.com/drive/folders/1ASUYXlWvKEzJ1VcfmYS7UOvhSTtlQors?usp=sharing
DevNet 6 experienced significant stability issues resulting in network failure. Multiple factors contributed to the problems:
- Bad blocks causing forking across all clients
- No client was able to stay synced even with itself, suggesting bugs across all implementations
- Root cause appears to be related to incomplete validator custody implementations
- Matthew explained that teams had to merge half-finished validator custody code when focus shifted to "getBlobsV2" and "getPayloadV5" implementation
- Derek mentioned the initial trigger may have been EL out-of-memory issues with Reth and Geth that cascaded to Nethermind
Gajinder raised important concerns about peer custody updates:
- Current Lodestar implementation uses static custody settings rather than dynamic ones
- When peer custody changes dynamically, there's no established protocol for notifying other peers
- Peers need to be informed to update their metadata when custody changes
- The team discussed whether metadata is updated with each ping or if additional mechanisms are needed
- This issue wasn't previously considered in the peerDAS planning discussions
The team discussed the relationship between getPayloadV5 implementation and DevNet 6 issues:
- The PR for getPayloadV5 is ready for final approval from Gajinder
- Gajinder clarified that in the current version, proofs from EL are discarded and replaced with self-computed proofs
- In getPayloadV5, the proofs sent from EL will be used directly
- This change relates to how cell proofs vs. blob proofs are handled
Nico initiated a discussion about test configuration standardization:
- Question about which presets (mainnet vs. minimal) to use in unit and end-to-end tests
- Nazar explained that only beacon node and state transition packages are still using mainnet preset for unit tests due to hardcoded fixtures
- Team consensus is to prefer minimal preset for faster tests where possible
- Noted concern that some unit tests take over a minute to run, which is excessive for developer feedback cycles
- Nazar will open an issue to address removing mainnet preset from unit tests
Phil noted that EPF applications are open:
- Encouraged directing promising contributors to apply
- Team can mentor EPF participants on Lodestar projects
- The beacon harness has been tagged as a potential EPF project
- Called for team members interested in mentoring to sign up and to tag any other projects of potential for EPF candidates
The upcoming Pectra fork is scheduled for May 7th:
- Will be early morning in North America, daytime in Europe
- Configuration for mainnet has been triple-checked
- Team is hoping for a smooth transition unlike some previous forks that required extensive debugging
Nico reported increased momentum for SSZ adoption in the Beacon API:
- Prysm is now pushing for full SSZ Beacon API implementation
- Discussion about making it mandatory in Fusaka
- Performance concerns with hex-encoded blobs driving the need for SSZ
- Lighthouse doesn't see JSON serialization as a significant issue
Nico:
- Critical vulnerability fix in consensus has been merged and is now public
- Made improvements based on NC's ideas using a map
- Fixed boot node issues identified by Nethermind on Chiado testnet
- Reviewed Electra-related PRs
- Participated in SSZ Beacon API discussions with other client teams
NC:
- Completed review of all state PRs, awaiting Nazar's feedback implementation
- Opened Blob parameter only fork work PR, discovered common unhandled corner cases with Derek's parallel work
- Updated weak subjectivity calculation for Electra
- Reported on FOCIL breakout meeting where Prysm and Lodestar are successfully interacting, while Teku continues having issues with incorrect root calculations
Katya:
- Completed validator monitor implementation (now merged)
- Working on validator monitor metrics and checks
- Investigating issues with ParaD monitor running locally
- Implemented validator activity metrics for active exits, consolidations, and withdrawals
- Applied script for Grafana dashboard updates but facing version compatibility issues
- Working on state transition metrics
- Plans to collaborate with Derek on pure DAS metrics[1]
Tuyen:
- Tested multi-functions implementation - all working except randomness due to WASM callback issue
- Continued work on pubkey index map implementation, using implementation from existing Zig code
- Fixed issue with handling zero block returns
- Fixed metric issue by tracking group index instead of group column index
- Working on additional metrics for range syncing
Nazar:
- Working on parallelizing block processing
- Current implementation processes block body attributes sequentially
- Opened PR with necessary refactoring to split implementation into smaller parallelizable functions
- Will address feedback on state diff PR based on NC's review
Derek:
- Planning to focus on testing this week
- Will combine writing tests with running DevNets to identify any unusual behaviors
Gajinder
- Reviewing PeerDAS PRs as requested by team members
- Investigating new exit mechanism proposed by Mike Neuter
- Phil noted the peerDAS branch is now running on Hoodi testnet and visible in Grafana under "nextfork" group for metric collection
Agenda: https://github.com/ChainSafe/lodestar/discussions/7699
Recording/Transcript: https://drive.google.com/drive/folders/1uvbbHIZlPYsFqRuz3N6K60A4Vt0Zj4Wi?usp=sharing
DevNet 6 deployment is progressing well with the team successfully getting nodes stood up and participating in the network. Matthew reported that the three major features required for DevNet 6 are either merged and deployed or ready to merge. However, there are some remaining implementation tasks:
- Hugh and Derek are working on updating validator custody implementation components, specifically:
- Updating peer registry to cache custody information from peers during status exchanges
- Updating network caches for optimized data fetching from appropriate peers
- Implementing ENR (Ethereum Node Record) updates when metadata changes
Tuyen noted peer stability issues in DevNet 6:
- Reporting low peer counts and connection failures: which is why we have hiccup issues sometimes getting out of sync and then getting back to normal.
- He observed that about 90% of disconnections were due to errors, and many peers were being ignored due to bad scoring.
Barnabas has reported that everything is going smoothly from the network perspective, despite these local peer issues[1].
The team engaged in a thorough discussion about improving the PR review workflow. Matthew proposed formalizing the review request process when PRs don't get sufficient attention:
- Developers should request specific reviewers through GitHub UI when PRs languish
- This would serve as a signal that a PR needs immediate attention without requiring centralized management
- Team members can decline reviews if they don't have the expertise or bandwidth
Nico expressed concerns about formalized assignment creating pressure to prioritize others' work over current tasks. Nazar suggested it's a human factor that helps ensure teamwork and prevents PRs from hanging.
The team reached consensus that:
- The current workflow works well for most situations
- Formal reviewer requests should be used sparingly when normal processes fail
- Monthly PR grooming will continue to address stale PRs
- The goal is to balance individual priorities with team needs
The team discussed preparations for version 1.29 release and Pectra fork readiness:
- Tuyen identified P2P issues as a potential blocker for 1.29
- Nico mentioned work on a PR for 1.29 that needs to be merged
- Phil assigned issue #7680 (producing block for build common block body while waiting for the builder) to Nazar
- The team noted timing may not be critical as mainnet releases would likely happen after the current development cycle
Nazar is working with NC on state diffing PRs, with a focus on breaking down the implementation into smaller, reviewable pieces.
Tuyen raised concerns about failing tests and linting issues in PeerDAS.
- Requesting team assistance to fix CI problems:
- Need more people to help with getting CI green because it's annoying to merge PRs.
- It doesn't require deep knowledge on PeerDAS to make this work.
The team acknowledged the importance of cleaning up CI before merging to unstable, with Phil encouraging everyone to contribute to this effort to make the process smoother.
Matthew shared an invitation from Soispoke for the team to participate in a security-focused "war mode" testnet:
- The initiative aims to simulate various attack scenarios to test network resilience
- Monthly attacks would test different security aspects (51% attacks, finality reversion, censorship, P2P DoS, etc.)
- The project appears to have originated from Vitalik's concern about "peacetime assumptions" in protocol design
- Some team members expressed interest, particularly Nico who mentioned this aligns with concerns since the Bellatrix issue
- Concerns were raised about publishing attack tools publicly, with suggestions to keep such capabilities in private forks
The team agreed to explore this opportunity further, particularly after current fork priorities are addressed.
Matt:
- Successfully deployed to DevNet 6 with all three major features ready
- Worked on refactoring, with first PR moving enums around and pushing branch for NEB classes
- Developing a roadmap for code reviews and merging of refactored components
- Identified remaining validator custody implementation needs with Derek and Hugh
Tuyen:
- Identified peer stability issues in DevNet 6 with low peer counts and connection failures
- We should cherry pick noble issue from unstable to peerdas to see if it helps with networking
- Working on implementing BLST-Z, achieving nearly 10x performance improvement in benchmarks
- Noted concerns about stability with the new implementation that requires further testing
- Planning to fix tests and implement reflective changes in the Bun side for BLST-Z release
NC
- Reviewed Tuyen's attestation packing PRs
- Shifted focus to hierarchical state diff research
- Studied Lighthouse implementation and reviewed Anton's hackmd notes
- Started reviewing PRs related to state diffing
Nico
- Reviewed other clients' attestation packing implementations compared to Tuyen's work
- Conducted preliminary analysis showing good results from the packing optimizations
- Working on MEV topics and researching EPBS implementation challenges
- Planning Ethereum guild discussions on MEV topics with potential guest speakers. Aiming for after Pectra hard fork.
Nazar
- Cleaned up linting and tooling-related issues
- Researched bun implementation differences compared to NodeJS
- Continuing work on state PRs based on NC's feedback
- Breaking down state feature into smaller, reviewable pieces to facilitate merging
Katya
- Had first PR merged
- Worked with Jim on server-side setups, restarting beacon node and setting up Grafana dashboard
- Currently working on validator monitor with Nico's assistance
- Launched local beacon node in testing mode (encountering hashing issues on Intel Mac that required workarounds)
- Note that hashtree upstream in Potuz's implementation doesn't support Darwin x86.
Gajinder
- Reviewed PRs on PeerDAS
- Commented on BPF4 configs and other specification-related matters
Agenda: https://github.com/ChainSafe/lodestar/discussions/7652
Recording/Transcript: https://drive.google.com/drive/folders/1ULDPYnf--vSKuGIJY8y0W9rs2H9lExnC?usp=sharing
Validator Custody PR
- Derek reported that the basic validator custody PR is ready to go, which implements dynamic custody group count without backfilling old columns
- The PR will include a flag to turn the feature on/off to prevent downscore issues since backfilling isn't implemented
- Matthew spent time reviewing the architecture and will finalize the PR review
DevNet 6 Requirements Clarification
- Gajinder provided important clarification about distributed blob building vs. current PeerDAS implementation
- Distributed blob building (where ELs distribute blobs based on node custody groups) is a future goal and not part of PeerDAS yet, as the spec isn't clear
- Two critical features needed for DevNet 6:
- Engine get_blobs_v2 API implementation
- Supporting cell proofs in transaction bundles
1. Purpose and Behavior of get_blobs_v2
- The
get_blobs_v2API is designed to allow Consensus Layer (CL) clients to fetch blobs from their local Execution Layer (EL) if the EL has all the required blobs for a block. This makes it a "hit-or-miss" feature:- If all blobs are available locally, the API provides them.
- If even one blob is missing, the response is empty.
- This behavior contrasts with distributed blob publishing, which is a future goal where ELs would aggressively distribute partial blob data (columns) based on custody groups. However, distributed blob publishing is not yet part of PeerDAS or DevNet 6 due to incomplete specifications.
2. Metrics and Data Collection
- The team emphasized the importance of collecting metrics to evaluate the effectiveness of
get_blobs_v2:- Metrics should measure how often
get_blobs_v2would succeed in real-world scenarios by tracking hit and miss rates. - Current metrics only show how many blobs were fetched but do not indicate whether all blobs for a block were locally available. This makes it challenging to assess the utility of
get_blobs_v2accurately.
- Metrics should measure how often
- A proposal was made to add a flag to enable metric collection specifically for this purpose. This would involve making additional calls to
get_blobs_v2, even when unnecessary, solely to gather data.
3. Incremental Implementation
- There was consensus that
get_blobs_v2does not need to be a blocker for DevNet 6 and can be added incrementally:- The team compared this approach to how
get_blobs_v1was introduced in stages. - The focus for DevNet 6 should remain on validating other critical features like cell proofs in transaction bundles and basic blob handling.
- The team compared this approach to how
4. Debate on Partial vs. Empty Responses
- A significant debate arose regarding whether
get_blobs_v2should return partial responses if some blobs are available or an empty response if all blobs are not present:- One perspective argued for empty responses to maintain simplicity and avoid confusion.
- Another perspective highlighted potential benefits of partial responses in scenarios like local block building, where some blobs might already be cached locally.
- Ultimately, it was noted that local block building typically involves importing blocks directly with all associated data already present, reducing the relevance of partial responses in this context.
5. Mainnet and DevNet Considerations
- For DevNet 6:
- It was agreed that EL clients should implement
get_blobs_v2, but CL clients could defer its usage depending on collected metrics.
- It was agreed that EL clients should implement
- For mainnet:
- The team recognized that
get_blobs_v2might become more critical as network conditions evolve and distributed blob publishing gains traction. - Differences in how various EL clients handle blob propagation will likely influence its utility across implementations.
- The team recognized that
6. Broader Implications
- The discussion highlighted broader challenges in PeerDAS development:
- Effective coordination between CL and EL clients is crucial for features like blob handling.
- Network topology, bandwidth constraints, and private vs. public mempools significantly impact blob availability and propagation rates.
- The team acknowledged that further research and testing are needed to refine these mechanisms before full-scale adoption.
While get_blobs_v2 is an important feature for PeerDAS, its immediate inclusion in DevNet 6 is not critical. The focus remains on collecting accurate metrics to inform its implementation and ensuring other foundational features are ready for testing.
- Team working toward releasing RC.0, targeting Thursday for the cut
- PRs currently merged into unstable branch
- Awaiting Tuyen's total effective balance sorting PR
- Waiting for Gnosis's configs, expected by the next day
- Additional potential PRs for consideration:
- Reducing maximum retain epoch from three to two for the seen attester cache
- Updating weak subjectivity calculation from the spec
- Team agreed these are not critical blockers but would be nice to include if ready by Thursday
Matthew
- Spent significant time reviewing architecture and PeerDAS implementation
- Got Tuyen's PR merged for global config
- Researched specs for
get_blobs_v2and cell publishing - Reviewed Nazar's PRs (~6 of them) focused on state diffs, linting, and JSR package management
- Made progress on refactoring code with approximately 1000 lines committed locally
- Focused on massage implementation and updating code to match newer paradigms
Derek & Hugh (Coinbase)
- Working on validator custody PR implementation
- Planning to move on to distributed blob publishing and engine get_blobs_v2 once the custody PR is merged
- Preparing for implementing cell proof support in transaction bundles
Katya
- Completed docs PR that's ready for review
- Finished validator monitor refactoring
- Running a beacon node (Hoodie) but having issues with metrics visibility in Grafana
- Will coordinate with Philip to resolve Grafana access issues
Tuyen
- Focused on packing attestation deployment and performance testing
- Deployed improvements over the weekend and gathered performance data
- Started work on multi INV implementation, aiming to finish within a week
- Encountered performance issues with BLST after SSZ 1.2 upgrade, particularly with hash computation
- Despite 30% faster performance on mainnet, holding off on further optimization until migration to Bun
Nico
- Helped merge PRs by Tuyen and Cayman, including IPNM (IP Number Management) implementation
- Spent time reviewing attestation-related code
- Tuyen's scoring PR looks promising
- Planning to coordinate with Luca for another data collection on Hoodie to compare with other clients
- Focusing on deeper understanding of Mev (Maximal Extractable Value) to prepare for EPBS implementation
- Reached out to Flashbots regarding MEV discussions, though they have limited time availability
NC
- Reported that Prysm and Lodestar fixed their consensus issue on FOCIL, now interoperating correctly
- Multiclient network is working well with minor hiccups to resolve
- Identified issue with Lodestar not publishing inclusion lists when slot time is set to 6 seconds
- Shifting focus back to EPBS (Ethereum Protocol Blockchain Staking) discussions
- Reviewing Tuyen's PRs and following attestation packing developments
Phil
- Working with Chiemerie on setting up the next fork group to track Electra fork progress
- This will help monitor for regressions when merging PurDAS branch
- Reminded about quarterly reviews deadline
- Provided information about Petra book feedback form for team contributions
Agenda: https://github.com/ChainSafe/lodestar/discussions/7554
- The team discussed the 1.28 RC currently running on beta.
- Metrics look good overall, but comparison is difficult due to Holesky testnet.
- Proposed approach: Deploy to canary validators on CIP to get mainnet metrics before wider release.
- An RC1 is needed with the latest fix for pre-Electra attestations, which addresses a regression from the past two months.
- Some optimizations on pending consolidation may be included in a future 1.29 release rather than 1.28.
- Chiemerie is working on migrating genesis validators from ETH Panda Ops to Hoodi servers.
- Goal is to migrate testnet infrastructure to Hoodi by end of the week.
- This will provide more stable metrics for collection.
- Discussion on how to approach reviewing and merging large feature branches like binary diffs.
- Consensus that an Architecture Decision Record (ADR) or similar high-level documentation would be helpful before implementation.
- Suggestion to focus on interfaces and overall architecture first, then implementation details.
- Agreement that big PRs into unstable are acceptable for formal review, but process could be improved.
- Nazar presented work on package manager and release management tool updates.
- Team decided to table major tooling changes until after Electra release to avoid disruption.
- Discussion on potential future adoption of Bun and compatibility with current tools.
- A developer from Base has offered to contribute to PeerDAS implementation.
- Team discussing how to best leverage this external help, possibly focusing on validator custody.
Matt
- Reviewed PRs with Nazar
- Discussed execution decoupling and range syncing ideas
- Planning Ethereum Guild session on execution decoupling
Katya
- Opened PRs needing review
- Working on data client metrics
- Planning calls with Tuyen and Nazar for onboarding
Cayman
- Published JSLipP2Quick, which is performing well
- Developed a Zig build tool, seeking feedback
Gajinder
- Worked on PeerDAS spec changes for validator custody and API endpoints
- Discussing static vs dynamic validator custody implementation
Tuyen
- Fixed issue with pending deposits processing time
- Syncing node from Genesis for BatchHash feature
- Improvements to BLST-Z library build process
Nico
- Created GitHub issues from notes
- Merged native compute implementation
- Addressed critical security report from a white hat
NC
- Updated FOCIL implementation for easier testing with Electra
- Working on lazy load pattern using generators for pending deposits and consolidation
- Updated block and attestation rewards to support Electra
Nazar
- Refactoring archive store
- Preparing to open PR for binary diff feature
The team also discussed improving their process for reviewing large architectural changes and welcoming new team member Katya.
Agenda: N/A
- The team discussed the approach for the next release after the Holesky rescue efforts.
- Initially considered a hotfix (1.27.2) but decided to aim for a full 1.28 release instead.
- Plan to merge bug fixes and new features from the Holesky rescue branch into unstable.
- Aim to have a 1.28 RC (Release Candidate) by the end of the week.
- Considering a potential 1.29 release before mainnet for additional features.
- Plan to restore infrastructure using the latest Holesky rescue image.
- Will test new features piece by piece before merging into unstable for 1.28.
- Discussed using unstable branch for test infrastructure once critical fixes are merged.
- Two priority fixes need to be merged to unstable:
- Tuyen mentioned improvements in state root computation time (now ~1 second on Holesky).
- Plans to push optimizations that could improve performance by 5x.
- Lodestar performed well during the Holesky rescue, showing stability and efficiency.
- Resource usage is significantly lower compared to other clients (12GB RAM vs 60-100GB).
- Lodestar was stable for two weeks and produced blocks when network participation was low.
- Currently the only client that can take an unfinalized checkpoint.
- Suggestion to focus on checking performance metrics on a stable Holesky network.
- Discussion about potentially using feed groups for normal testing.
- Consideration of keeping a 300-validator key node for comparison with mainnet Lido validators.
Tuyen
- Coordinated with Nico on Holesky Rescue.
- Worked on BLST-Bun, completed spec tests and benchmarks.
- Found performance issues on Linux (3x slower than BLST-TS), suspecting build optimization problems.
- Requested Nico's help with the remaining work on unfinalized state feature.
Gajinder
- Discussed uncoupled execution payload with Potuz.
- Explored potential complexities with delayed availability of data blobs.
- Worked on a simplified version of PeerDAS validator custody architecture.
- Presented the simplified version in the PeerDAS breakout meeting.
Cayman
- Updated js-libp2p-quic repo, an experimental napi-rs wrapper for QUIC library.
- Helped with Holesky Rescue efforts.
- Worked with Nico to spec out ERA file support for archive node functionality.
- Continuing work on Zig build tooling CLI.
Katya
- Working on reorganizing metrics.
- Focusing on the state transition package.
- Learning the codebase and asking questions.
Nazar
- Has multiple PRs pending review, including one for erasable syntax.
- Preparing to publish the first package on JSR, starting with the benchmark package.
- Working on modernizing the publishing pipeline to support various ecosystems.
Nico
- Led efforts in Holesky Rescue, fixing critical issues.
- Unfortunate that we had the attestation bug, but we performed very well comparatively.
- Resource usage was stable at around 12GB.
- We were stable for two weeks and still produced blocks.
- We were stable even throughout the 5% participation where the network almost collapsed.
- We pointed 100,000 validator keys to one beacon node and still performed well.
- Experimented with loading 100,000 validator keys in a single validator client.
- Possible to load, the validator client had a 16GB heap memory usage
- Goes into a death spiral though for initializing the slashing DB for all 100k validators
- Accidentally caused some validators to be slashed during key movements due to moving keys around
- Emphasized Lodestar's strong performance and reliability during the rescue efforts.
The team concluded the meeting by discussing the next steps for stabilizing the network and returning to normal development rhythms. They also mentioned upcoming opportunities for conference presentations and travel arrangements.
Agenda: https://github.com/ChainSafe/lodestar/discussions/7524
- Discussed pruning previous states and rejecting blocks before a certain depth
- Considered using an epoch state at a particular depth as the finalized state, similar to non-finalized checkpoint start
- Tuyen's PR to delete after 100 epochs was mentioned as an example of current practice on Holesky fork. We won't be able to regen beyond 100 epochs ago
- Concerns raised about range sync always starting from the last finalized state, potentially causing issues with long non-finalized chains
- Lodestar experiencing issues with continuous batch generation from last finalized state during long periods of non-finalization
- Discussion on why nodes might be re-querying batches and potentially getting banned by peers
- Speculation on how Lodestar handles sync when connecting to new peers
- Current Holesky prod nodes are functioning, but testnet infrastructure is unstable
- Need for testing infrastructure for 1.28 release and future updates
- Challenges with testing on Sepolia due to low validator count (around 1700)
- Consideration of using CIP validators for beta testing
- Plan to have 1.28.0-RC by the end of the week
- Discussion on which PRs to include, such as peer manager updates and blacklisted block check
- Agreement to release an RC with confident changes and deploy to CIP validators for sanity checking
Matt
- Continued work on a large code change, writing another thousand lines
- Preparing to break down the changes into PR-able pieces
- Reviewed Nazar's work and preparing for state diff transition
Nazar
- Merged several PRs, working on renaming archive store
- Preparing binary diff PR for review
- Added support for JSR registry publishing in SSZ repo
- Investigating Node erasable syntax for potential future TypeScript execution in Node.js
Tuyen
- Fixed option and historical state file data store issues
- Working on syncing from unfinalized checkpoints
- Progress on BLST in Bun, aiming for first release next week
Katya
- Started work on metrics, focusing on beacon metrics specs
- Found some unused specs and planning to push initial PR after review
Nico
- Discussed pessimistic sync implementation challenges
- Planning to test Tuyen's PR for syncing from unfinalized states
- Considering testing NC's pessimistic sync on Holesky
Gajinder
- Proposed an EIP to uncouple execution payload from BeaconBlock
- Discussed potential benefits of this approach compared to EPBS
- Suggested a dedicated session to discuss the EIP in detail
The team also discussed plans for an upcoming Ethereum Guild meeting to debate various topics, including Gajinder's EIP proposal.
Agenda: N/A
- The team is nearing readiness for the Pectra Hard Fork scheduled for next Monday.
- Holesky Genesis validators and Sepolia groups are updated and ready. Awaiting mev-boost for updating
- Execution Layer (EL) clients connected to testnet infrastructure are being upgraded.
- Feature branches need to be rebased to v1.27 and redeployed to ensure proper forking. If not completed, version
v1.27will be deployed as a fallback at the end of the week
- A potential consensus issue was reported by Gnosis, but its root cause remains unclear. It may relate to differences in presets or hardcoded constants between mainnet and Gnosis chain.
- A specific Lodestar node showed a state root mismatch during state transitions one Gnosis node only, while others were unaffected.
- Suspected causes include caching issues or discrepancies in pre-state data. It was noted that states are different from the problem node and the healthy node. Slot 68,120 it seems like it has a different pre-state in the bad node.
- Further investigation is required, including downloading API data and comparing states.
- https://github.com/ChainSafe/lodestar/issues/7299
- Sync aggregate problems were observed on DevNet 6 and potentially Holsky/Mainnet.
- Some blocks failed due to delayed parent block visibility of greater than 6 seconds or other unknown factors.
- The team will continue monitoring and investigating these issues.
- DevNet is being restarted due to issues with the Geth client. Fixes are pending release.
- Optimistic syncing was identified as a cause for nodes appearing synced without validator attestations. This issue should resolve once EL fixes are applied.
- Matt continues working on refactor here: https://github.com/ChainSafe/lodestar/pull/7474
- SSZ repository has been added to ChainSafe with admin access pending setup for Lodestar teams: https://github.com/ChainSafe/ssz-z
- SSZ operations would benefit from being lightweight and fast in the native implementation.
- The design allows for seamless data retrieval and caching between native and JavaScript layers.
- A proposal for modernizing Lodestar using Zig was discussed:
- Focus on native state transitions leveraging Zig's lightweight pointer management. A proposed mechanism involves storing pointers as numeric identifiers in JavaScript, which can be passed back to the native side for operations.
- Potential benefits include faster data access and shared memory between native and JavaScript environments.
- Concerns about memory space limitations (e.g., U32 vs. U64) were addressed with existing safeguards in place.
- Native State Transition: This approach aims to improve performance and memory efficiency by bypassing JavaScript's limitations.
- Pointer-Based Architecture:
- A pointer in Zig is represented as an 8-byte value.
- Lodestar's architecture would maintain a "BeaconState" pointer, which can be passed to native functions for state transitions, reducing overhead.
- Shared Data Access: Both native and JavaScript environments can access shared data structures, minimizing duplication and improving performance.
- While some features may depend on unstable Zig APIs, the design ensures compatibility with existing tools and safeguards against potential limitations. It will require careful monitoring and updates as the language evolves.
Next Steps:
- Begin prototyping the state transition module in Zig.
- Evaluate performance improvements through benchmarks.
- Address integration challenges with existing Lodestar components (e.g., SSZ serialization).
- Plan resource allocation, including potential team expansion, to support this roadmap.
- Discussion on whether Docker and non-Docker deployments should be maintained across all testing groups:
- Docker deployments generally perform better due to up-to-date system libraries (e.g., glibc).
- Most users deploy via Docker, making it essential for testing.
- Decision: Maintain both Docker and binary testing but simplify where possible.
- Plans to hire additional engineers to alleviate workload and support the Zig-native roadmap. Referrals are encouraged.
Cayman:
- Developed a native version of sync committee computation using NAPI-RS, achieving a ~40x performance improvement (10β15ms computation time).
- Worked on SSZ refactoring; seeking reviewers for progress discussion.
Matt
- Progressed on a major refactor PR, addressing errors in sync logic and range sync. The draft PR is available for preliminary review.
Nazar
- Four pull requests are under review; feedback will be incorporated this week despite health challenges.
Nico
- Updated fork testing from Altair/Bellatrix to Electra payloads, resolving spec issues in the Beacon API and Builder API.
- Released a major Beacon API update validated against Lodestar.
- Continued debugging interop issues, particularly with Prism clients.
Tuyen
- Investigated sync committee performance bottlenecks; native implementations show promising improvements.
- Identified performance issues in PeerDAS deployments due to lack of unit tests; ongoing refactor expected to address this.
- Explored callback integration for FFI but deemed it lower priority compared to other tasks.
NC:
- Fixed block attestations issue caught on devnet-6
- Read about ethspecify and investigate its potential benefit for Lodestar see #7477
- Focil - caught a Prysm bug that its
state.forkis not updated correctly on focil fork, which is one of the causes of state root mismatch I mentioned last week. - Lodestar also has fork digest issue which I had fixed
- Regarding running a stress test of having large number of deposit requests β Electra Discussionsβ I have pinged pk910, hopefully we can get some result soon.
Agenda: https://github.com/ChainSafe/lodestar/discussions/7436
- The team is preparing for the Pectra testnet release, scheduled for Thursday.
- Several PRs (e.g., Snappy and Async Aggregate) will be reverted due to unresolved issues.
- The
pruneHistoryfeature will not be reverted but hidden via a flag as it is not yet production-ready. Concerns were raised about its behavior during restarts, particularly with LevelDB compaction delays.
- The Pectra release will focus on testnets and include hard fork numbers for Sepolia and Holesky.
- A mainnet release date will be decided after the Sepolia fork, with expectations for one or two more releases before the official Pectra mainnet release.
- Discussions highlighted whether to integrate the PeerDAS (Fulu) code into the Pectra release. The consensus was to delay merging large changes until after mainnet to ensure stability and thorough testing.
- Validator custody complexities were debated, with concerns about its implementation difficulty and enforcement. Suggestions included manually increasing custody data for nodes with more validators.
- The role of super nodes in PeerDAS was discussed. While they can enhance network performance, their necessity for network stability was questioned. It was emphasized that availability should not hinge on super nodes alone.
- Preliminary discussions on Lodestar's involvement in the Beam chain highlighted ongoing work in cryptographic advancements (e.g., post-quantum signatures) and modular architecture to support future proof systems.
- The Fabric initiative, focusing on L2 standardization (e.g., bridging, DA layers), was introduced as a potential direction for Lodestar's modular architecture.
Matt
- Progressed on PeerDAS integration:
- Spec tests, unit tests, and linting are nearly complete.
- PeerDAS branch is synced on DevNet.
- Proposed renaming fork-related types (e.g., Fork Blobs) in line with spec conventions (e.g., pre-Deneb/post-Deneb).
Nico
- Reviewed remaining Electra tasks; identified an attestation/block production issue on DevNet.
- Beacon API cleanup is ongoing; deprecated the deposit API following community consensus.
NC
- Focused on Fossil development:
- Resolved Engine API issues with Geth.
- Ran local devnets with Lodestar-only setups, achieving finalized inclusion lists.
- Encountered state root mismatches when testing with Prysm; debugging continues.
Tuyen
- Worked on Proposal Boost reorg fixes; tests passed but sync tests need refinement.
- Released BLST-Zig integration using Zig compiler for BLST bindings.
- Addressed PeerDAS issues, including ensuring minimum match pieces per group.
Cayman
- Finalized
pruneHistoryflag implementation but will hide it due to rough edges. - Reverted Snappy and Async Aggregate PRs due to regressions when combined; plans to test individually moving forward.
- Continued SSZ-Zig refactoring; nearing completion despite challenges from ongoing updates.
Gajinder
- Highlighted critical issues in delayed block proposals:
- Suggested upfront transaction fee deductions to prevent invalid blocks from impacting coinbase payouts under deferred execution models.
- Advocated delaying Fossil until delayed execution is fully tested.
- Engaged in discussions on validator custody and super node roles within PeerDAS.
Helena
- Announced an upcoming blog post on censorship resistance by Ben and Phil.
- Encouraged team members to submit speaking proposals for Protocol Guild and ECC events.
Action Items
- Finalize Pectra testnet release by Thursday, ensuring reverted PRs are excluded or flagged appropriately.
- Investigate Electra attestation/block production issues further before testnet deployment.
- Continue debugging Prysm state root mismatches for Fossil compatibility testing.
- Prepare a unified message for Lodestarβs stance and contributions to the Beam chain initiative ahead of Fridayβs call.
Agenda: https://github.com/ChainSafe/lodestar/discussions/7422
- RC1 was pushed yesterday with a change from Nico to increase block production timeouts
- No issues reported so far on beta deployment and Nico's mainnet validators
- Team will do a final review after the call before releasing
- Discussion on optimizing testing infrastructure and server configurations
- Focus on covering both Docker and binary setups due to performance differences
- Proposal to have ARM64 servers in unstable, stable, and beta groups for sanity checking, not required in feature groups.
- We should make sure that all servers of a similar type are actually using similar hardware.
- Redunancies to the core use case/workload that we measure against is important
- Consider that subscribe all subnets helps spot regressions more than it is to be performant while subscribing to all subnets
- We should have holesky validators attached with subscribe all subnets, same hardware to do better apples to apples comparisons
- Debate on the usefulness and performance impact of subscribe all subnets feature
- Consensus that it's not beneficial for validator effectiveness but useful for testing
- We generally see performance of validators suffering under it and is not really beneficial at all.
- Head votes are worse
- Block imports are delayed
- A worse peer overall
- Might be worth putting in our documentation that subscribe all subnets is not recommended for normal use in any validator staking configuration
- We've come to the conclusion that some changes improve validator performance but may regress metrics on subscribe all subnets. There is no correlation that shows better performance on subscribe all subnets means better validator effectiveness for normal users.
- Decision to use it as a tool for spotting regressions and optimizations, but not to cater resources specifically to being performant with subscribe all subnets
- Agreement to not advertise or advocate for subscribe all subnets as a primary feature
- Snappy-wasm + async aggregate will be rebased and tested on a feature server for inclusion
- JS-libp2p will also be rebased and tested on a feature server for inclusion
- Matt will also re-test Multi-Scalar Multiplication (MSM) functionality for blst-ts as an independent feature on unstable
- PeerDAS devnets are turning into Fulu devnets, additional PRs going in to ensure readiness for the next devnet, including Tuyen's optimizations:
- Expected to be ready for review soon
- This is a breaking change but will be hidden under a flag for merging to unstable and disabled by default
- It is not backwards compatible once upgraded
- Once we test everything and the feature is matured, we can then think about writing a migration to completely deprecate and remove the old format which is more important for historical nodes rather than validators
- It is also possible to just use a different bucket name to prevent it from being a breaking change
- Consideration for making it default in Lodestar 2.0 after thorough testing where we can see the reduction of disk size usage
- Note that computation will likely be higher for generating binary diffs
- Discussed having a pruning history flag to get rid of historical blocks and states (we currently store one state every 4096 slots by default) for better UX.
- Some validator type users can have up to 500GB of worthless storage that can be cleaned up via startup using this flag to purge history
- https://github.com/ChainSafe/lodestar/pull/7427
- Discussion on managing long-lived fork branches
- Agreement to merge sooner into unstable branch to avoid conflicts
- Emphasis on merging code that's already in consensus spec into unstable
NC
- Working on FOCIL implementation
- Encountering issues with Engine API and syncing
- Planning to focus efforts on resolving these issues before the next FOCIL call
Nazar
- Binary diff PR nearly ready for review
- Will deploy to feature group for testing after review
Matt
- Completed work on PeerDAS
- Planning to look into MSM implementation
- Working on refactoring PeerDAS code and implementing Twiyn's suggestions
- Discussion on deprecating deposit snapshot after Pectra goes live
- Consideration of pruning old blocks for improved storage management
- Ongoing work on various PRs including flattening fork logic and deprecating eth1 data poll
Agenda: https://github.com/ChainSafe/lodestar/discussions/7395
- Version 1.26 is scheduled for RC release on Friday.
- DevNet 6 is expected to happen this week, but the timeline is uncertain as teams are still fixing DevNet 5 bugs.
- The team aims to include some hard fork numbers by Thursday's ACD call.
- SSZ 1.0.2 release is facing CI issues, but the team plans to swap the token and try again.
- Most remaining Electra tasks are optimizations and not urgent for the 1.26 release.
- The team wants to include the Electra attester slashing handling in Friday's release.
- The team discussed the challenges faced during the Petra upgrade and how to improve for future hard forks like Fulu.
- Key issues identified:
- Scope creep and difficulty in maintaining timelines
- Challenges in prioritizing features and EIPs
- The need for better communication and consensus-building mechanisms
- Suggestions for improvement:
- Consider implementing a more structured approach to EIP inclusion and hard fork planning
- Potentially increase the frequency of hard forks (e.g., two per year) with smaller scopes
- Improve tooling and processes for identifying consensus and prioritizing features
- Balance between decentralized decision-making and the need for clear direction
- The team discussed the difficulty of removing features from a hard fork once implementation has begun, as they are often intertwined in the codebase.
- There was a discussion about the potential benefits of a more modular architecture that would allow for easier feature toggling, similar to what some other clients have implemented.
- The importance of maintaining Ethereum's high reliability standards was emphasized, with the team agreeing that shortening hard fork cycles should not compromise security or stability.
- The team is working on fixing CI issues for the SSZ 1.0.2 release.
- Work continues on optimizing and finalizing Electra features for the upcoming 1.26 release.
- The team is closely monitoring DevNet 5 and preparing for DevNet 6.
- There was a discussion about recent developments in the L2 ecosystem, particularly regarding the Fabric initiative and its potential impact on Ethereum's development process.
The team will continue to work on the 1.26 release and prepare for upcoming discussions at the ACD call and town hall meeting.
Agenda: https://github.com/ChainSafe/lodestar/discussions/7360
- The team discussed the roadmap for Lodestar V2, reviewing a pinned issue with the planned milestones.
- There was a debate about whether to support old forks in V2:
- Consensus was to keep old forks for now unless they become blockers, as removing them would require significant effort.
- The team agreed to remove the sim merge test, as pre-merge networks are no longer relevant.
- Some regressions were observed after merging DevNet 5 to Unstable:
- Increased job wait times
- Issues with mesh peers
- Higher GC on the main thread
- Increased attestation reprocessing (from 4 to 24 per slot on average)
- Gajinder reviewed the new EIP for delayed state root computation.
- Concerns were raised about potential complications for CL validations:
- Without the state root, it's harder to verify if a block was properly executed.
- Suggested adding a commitment or hash of state diff changes to address this issue.
- The scheduled RC release this week may be pushed to next week, depending on the progress in fixing the regressions on unstable and to accommodate potential hard fork release epochs for public testnets.
Matthew:
- Merged non-SIMD changes
- Worked on PeerDAS integration with DevNet 5
- Reviewing PeerDAS specs for discussion with Gajinder
Nazar:
- Fixed memory leak in benchmark setup caused by Vitest runner
- Working on migrating Lodestar repo benchmarks to the upgraded version
Cayman:
- Worked on a PR for Tuyen's ZIG SSZ library (paused for now)
- Analyzed metrics for libP2P 2.0 branch (feat-4), looking promising
- Started familiarizing with sync code for potential refactoring or backfill sync work
- Planning to revisit the Bun runtime effort for running Lodestar
Gajinder:
- Assisted with Kurtosis configs debugging
- Investigated NAPRS and TypeScript type interactions
- Reviewed and commented on EIP for delayed state root (EIP 7732)
Nazar:
- Fixed memory leak in benchmark utility related to Vitest runner
- Working on migrating Lodestar repo benchmarks
Andrew:
- Reaching out to node operators for feedback
- Potential new node operator interested in testing Lodestar
Nico:
- Discovered issues with some testing servers not running binaries as expected
- Planning to rejig testnet infrastructure, consolidating smaller validator nodes
- Participated in Twitter spaces event bridging builders with core devs and researchers
The team will continue to debug the Unstable branch issues asynchronously and reassess the RC release schedule based on progress.
Agenda: https://github.com/ChainSafe/lodestar/discussions/7337
- Metrics review for 1.25 release showed no concerning issues. Release planned for later today.
- Next RC due date discussed for 1.26:
- Agreed to cut RC on January 24th, aiming for release on January 31st
- Acknowledged shorter timeframe but aligns with goal of more frequent releases
- Plans include merging work to switch to Snappy Wasm for decompression once it is rebased and redeployed.
- They also want to revisit the libp2p 2.0 and GossipSub updates, testing them separately from the βIDONTWANTβ feature to address prior latency issues.
- SIMD support for SSZ was merged and tested, aiming for inclusion in 1.26
- Continued benchmarks for different JavaScript runtimes (e.g., Bun, Node) are underway to handle garbage collection impacts on benchmarking consistency
The discussion around the Devnet-5 branch cleanup and integration plan back into unstable covered a few key points:
- There is a need to clean up the DevNet-5 branch.
- The branch contains changes that are not only related to Electra but also affect how forks are handled.
- The DevNet-5 branch includes changes to the NEP (Network Upgrade Process), which requires caution during merging.
- There are preset value updates in the minimal preset that affect CI and sim tests.
- Current CI is failing due to the preset changes.
- The team needs to wait for a Lighthouse image with updated presets to fix the CI issues.
- As a temporary measure, they might consider running tests up to Deneb instead of Electra until the Lighthouse image is available.
Action Items
- Review and fix any issues in the DevNet-5 branch before merging.
- Wait for the Lighthouse image update to resolve CI problems.
- Plan the integration carefully to avoid disrupting the main development flow.
The team emphasized the importance of not rushing the integration process and ensuring thorough testing before merging DevNet-5 changes into the unstable branch.
- Team to review and tag remaining Electra-related todos
- Plan to integrate DevNet-5 changes into Unstable after 1.26 RC
Gajinder
- Reviewed latest PeerDAS spec changes
- Investigated TypeScript and Rust interop for potential light client snarkification
Nazar
- Working on moving runtime setup to SSZ repo
- Debugging benchmark issues related to garbage collection
Nico
- Completed gas limit check PR
- Finished remaining DevNet-5 tasks
- Started looking into FOSSIL implementation
Matt
- Finalized remaining work on the SSZ SIMD PR.
- Focused on PeerDAS groups PR and a thorough review of the overall PeerDAS code.
- Rebasing changes on Electra and merging updates for Fulu, aiming to test them in DevNet soon.
Cayman
- Made PRs to discv5 for library improvements and removing costly crypto dependencies.
- Reviewed the SSZ ZIG implementation and has a PR coming for enhancements.
- Continues to review PRs such as Peer Discovery for subnet requests and related code[2].
Tuyen
- Continued work on BLST binding for Zig and Bun testing signature, secret key, and public key classes.
- Working on more advanced features like verifying multiple aggregate signatures.
- Raised small PRs for PeerDAS improvements, including minimum subnet checks and batch download fixes.
NC
- Focused on FOCIL implementation, estimated to be halfway complete.
- Contributing to the Beacon API aspects for FOCIL and coordinating on design discussions in the open spec documents.
Phil
- Upcoming call with Alex Stokes and Peter for quarterly feedback
- Coordinating team attendance for upcoming interops
Andrew
- Followed up on RPC endpoints discussion with infrastructure team
- Reconnecting with operators for feedback, including questions on new release stability and validator effectiveness data
Peter
- Highlighted an upcoming blog post draft for a new βFabricβ standard for rollups, aiming to unify L2 solutions without introducing new networks or tokens
- Mentioned synergy with potential L2 API expansions in Lodestar and continuing research on universal rollup definitions
Agenda: https://github.com/ChainSafe/lodestar/discussions/7326
- Implement stricter cadence for releases to prevent performance regressions from large accumulations of unreleased commits.
- Proposed cadence: Close door to PRs on specific day (Friday), cut RC, deploy to beta group, review metrics and release after standup the following week.
- Goal is to avoid situations like the recent accumulation of 97 PRs.
-
Various PRs discussed. The following aimed for this specific release:
- Snappy Wasm to be redeployed, observed and merged to unstable
- refactor: update snappy frame decompress to be included
- feat: use subnet request mechanism for subnet sampling strategy to be reviewed with Tuyen and Matt
- refactor: use SubnetID type from CL spec instead of number has merge conflicts. To be resolved by Nico
- feat: implement EIP-7691 increase blob throughput needs one more round of reviews. Needs to be merged for devnet-5 launch
- Confirmed two-week cadence for releases; aim to deploy RC to beta group by end of week.
- FOCIL project gaining momentum; need to assign lead and break down spec for implementation.
- NC and Nico will take charge on this implementation
- Spec itself is small. Changes to beacon node, validator, P2P and fork choice. Due for next breakout call January 15.
- PeerDAS is high priority; need to get it working to avoid max blob count issues. Continue implementation and testing.
- Matt and Gajinder will take charge on this implementation. Tuyen in support role.
- Kurtosis configs and interop testing scheduled for two weeks from now.
- Local testing via Kurtosis is ongoing, but no public spin-up yet.
- Builder flow will start being tested.
- Focus on fixing bugs and ensuring stability before public release.
- Closing role, shifting BlockOps role back towards software engineering focus.
-
Cayman:
- Completed ENR encoding, decoding, signing, and verifying implementation in Zig.
- Will share work and findings with team soon.
-
Tuyen:
- Making progress on getting Zig BLS library operational.
- Expects to have good news to share next week.