ci(levm): silence messages #1837

fborello-lambda · 2025-01-29T19:25:42Z

Motivation

We would like to suppress certain messages and run the hive tests in the ci_levm.yaml file.

Description

Silence LEVM EF-TESTS messages if all of them pass.

Send a message to slack only if we don't have a 100%
Create the PR comment with the Summary only if we don't have a 100%
- If we create the comment, update it later with the message All EF-TESTS pass. if the bug is fixed within the PR.

Silence LEVM Hive Tests messages if no differences were found with respect to revm.

Add Hive Tests check to LEVM CI → fails if differences were found
Send a message to slack only if we do find differences.

github-actions · 2025-01-29T19:28:37Z

| File                                                                                            | Lines | Diff |
+-------------------------------------------------------------------------------------------------+-------+------+
| /home/runner/work/ethrex/ethrex/cmd/ef_tests/levm/runner/revm_runner.rs                         | 462   | +2   |
+-------------------------------------------------------------------------------------------------+-------+------+
| /home/runner/work/ethrex/ethrex/crates/blockchain/blockchain.rs                                 | 185   | +3   |
+-------------------------------------------------------------------------------------------------+-------+------+
| /home/runner/work/ethrex/ethrex/crates/blockchain/constants.rs                                  | 18    | +6   |
+-------------------------------------------------------------------------------------------------+-------+------+
| /home/runner/work/ethrex/ethrex/crates/blockchain/error.rs                                      | 93    | -2   |
+-------------------------------------------------------------------------------------------------+-------+------+
| /home/runner/work/ethrex/ethrex/crates/blockchain/payload.rs                                    | 481   | -18  |
+-------------------------------------------------------------------------------------------------+-------+------+
| /home/runner/work/ethrex/ethrex/crates/common/core.rs                                           | 4     | -1   |
+-------------------------------------------------------------------------------------------------+-------+------+
| /home/runner/work/ethrex/ethrex/crates/common/types/block.rs                                    | 643   | -4   |
+-------------------------------------------------------------------------------------------------+-------+------+
| /home/runner/work/ethrex/ethrex/crates/common/types/genesis.rs                                  | 418   | -197 |
+-------------------------------------------------------------------------------------------------+-------+------+
| /home/runner/work/ethrex/ethrex/crates/networking/p2p/sync.rs                                   | 795   | -9   |
+-------------------------------------------------------------------------------------------------+-------+------+
| /home/runner/work/ethrex/ethrex/crates/networking/rpc/engine/payload.rs                         | 445   | -4   |
+-------------------------------------------------------------------------------------------------+-------+------+
| /home/runner/work/ethrex/ethrex/crates/networking/rpc/eth/block.rs                              | 358   | -12  |
+-------------------------------------------------------------------------------------------------+-------+------+
| /home/runner/work/ethrex/ethrex/crates/networking/rpc/eth/fee_market.rs                         | 178   | -20  |
+-------------------------------------------------------------------------------------------------+-------+------+
| /home/runner/work/ethrex/ethrex/crates/vm/levm/src/call_frame.rs                                | 124   | +6   |
+-------------------------------------------------------------------------------------------------+-------+------+
| /home/runner/work/ethrex/ethrex/crates/vm/levm/src/errors.rs                                    | 247   | +8   |
+-------------------------------------------------------------------------------------------------+-------+------+
| /home/runner/work/ethrex/ethrex/crates/vm/levm/src/execution_handlers.rs                        | 251   | +13  |
+-------------------------------------------------------------------------------------------------+-------+------+
| /home/runner/work/ethrex/ethrex/crates/vm/levm/src/opcode_handlers/push.rs                      | 62    | -2   |
+-------------------------------------------------------------------------------------------------+-------+------+
| /home/runner/work/ethrex/ethrex/crates/vm/levm/src/opcode_handlers/stack_memory_storage_flow.rs | 270   | -1   |
+-------------------------------------------------------------------------------------------------+-------+------+
| /home/runner/work/ethrex/ethrex/crates/vm/levm/src/vm.rs                                        | 718   | -1   |
+-------------------------------------------------------------------------------------------------+-------+------+

Total lines added: +38
Total lines removed: 271
Total lines changed: 309

github-actions · 2025-01-29T19:38:21Z

Benchmark Results Comparison

PR Results

Benchmark Results: Factorial

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
`revm_Factorial`	234.9 ± 0.6	234.2	236.1	1.00
`levm_Factorial`	904.2 ± 6.8	894.2	916.8	3.85 ± 0.03

Benchmark Results: Factorial - Recursive

Command	Mean [s]	Min [s]	Max [s]	Relative
`revm_FactorialRecursive`	1.496 ± 0.070	1.370	1.573	1.00
`levm_FactorialRecursive`	15.793 ± 0.043	15.726	15.868	10.56 ± 0.49

Benchmark Results: Fibonacci

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
`revm_Fibonacci`	213.5 ± 3.3	208.4	220.5	1.00
`levm_Fibonacci`	910.7 ± 10.9	901.0	937.9	4.27 ± 0.08

Benchmark Results: ManyHashes

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
`revm_ManyHashes`	8.7 ± 0.1	8.6	8.8	1.00
`levm_ManyHashes`	18.1 ± 0.1	18.0	18.2	2.09 ± 0.02

Benchmark Results: ERC20 - Transfer

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
`revm_ERC20Transfer`	258.8 ± 4.5	250.1	262.2	1.00
`levm_ERC20Transfer`	3224.9 ± 16.4	3195.2	3241.5	12.46 ± 0.23

Benchmark Results: ERC20 - Mint

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
`revm_ERC20Mint`	144.3 ± 0.6	143.8	145.9	1.00
`levm_ERC20Mint`	1694.3 ± 10.0	1678.5	1706.2	11.74 ± 0.09

Benchmark Results: ERC20 - Approval

Command	Mean [s]	Min [s]	Max [s]	Relative
`revm_ERC20Approval`	1.042 ± 0.004	1.036	1.049	1.00
`levm_ERC20Approval`	11.420 ± 0.084	11.280	11.531	10.96 ± 0.09

Main Results

Benchmark Results: Factorial

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
`revm_Factorial`	239.5 ± 1.6	238.2	243.4	1.00
`levm_Factorial`	908.5 ± 6.2	902.4	924.2	3.79 ± 0.04

Benchmark Results: Factorial - Recursive

Command	Mean [s]	Min [s]	Max [s]	Relative
`revm_FactorialRecursive`	1.497 ± 0.082	1.422	1.682	1.00
`levm_FactorialRecursive`	15.684 ± 0.022	15.660	15.726	10.48 ± 0.57

Benchmark Results: Fibonacci

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
`revm_Fibonacci`	208.6 ± 0.8	207.3	210.1	1.00
`levm_Fibonacci`	914.3 ± 8.9	902.0	930.6	4.38 ± 0.05

Benchmark Results: ManyHashes

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
`revm_ManyHashes`	8.7 ± 0.1	8.6	8.7	1.00
`levm_ManyHashes`	18.3 ± 0.3	18.0	18.9	2.11 ± 0.03

Benchmark Results: ERC20 - Transfer

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
`revm_ERC20Transfer`	252.9 ± 1.7	251.4	257.0	1.00
`levm_ERC20Transfer`	3222.8 ± 27.0	3187.3	3263.7	12.74 ± 0.14

Benchmark Results: ERC20 - Mint

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
`revm_ERC20Mint`	144.2 ± 0.9	143.3	145.7	1.00
`levm_ERC20Mint`	1690.6 ± 13.1	1674.4	1703.1	11.73 ± 0.11

Benchmark Results: ERC20 - Approval

Command	Mean [s]	Min [s]	Max [s]	Relative
`revm_ERC20Approval`	1.058 ± 0.010	1.049	1.083	1.00
`levm_ERC20Approval`	11.390 ± 0.075	11.293	11.484	10.77 ± 0.13

github-actions · 2025-01-29T20:40:39Z

Summary: All EF-TESTS pass.

mpaulucci · 2025-01-30T15:53:02Z

.github/workflows/ci_bench_levm_in_pr.yaml

@@ -2,7 +2,7 @@ name: Benchmark LEVM vs REVM in PR



why is this file not in ci_levm? That would be more consistent

Because one file has the Integration Test job, making it a required check.
And the ci_bench_levm_in_pr.yaml only modifies the github comments.

The logic was separated in case of an external contribution, if we send the PR comment inside the Integration Test job in order to avoid repetition, the job will fail because the external contributor doesn't have access to the TOKENs

mpaulucci · 2025-01-30T17:27:38Z

.github/workflows/ci_levm.yaml

+
+      - name: Check Regression
+        run: |
+          if grep -q "No differences found" diff.md; then


This doesn't seem to check that levm has a regresion (levm vs levm), but that levm is equal to revm. So this check will fail until there is feature parity between revm and levm. Is that what we want?

We would have to check against main, i will start with it, maybe it's too cumbersome for now. When LEVM is used as default we would definitely want the (levm vs levm) regression check.

I thinks it's done in commit e91040b. I save the daily artifact and use it for the comparison/check.

mpaulucci · 2025-01-30T17:28:52Z

.github/workflows/ci_levm.yaml

+  # Check we don't have a regression
+  hive-test-check:
+    # "Integration Test" is a required check, don't change the name
+    name: Integration Test


We have two integration test, can we do something like this instead? https://github.com/lambdaclass/ethrex/blob/main/.github/workflows/ci_l1.yaml#L207-L226

Working on it

Done in 5f3a652

.github/workflows/common_hive_reports.yaml

mpaulucci · 2025-01-30T17:36:21Z

.github/workflows/common_hive_reports.yaml

@@ -0,0 +1,100 @@
+name: Run Hive Tests
+
+on: workflow_call


would it make sense to have an input here that is vm?

See: https://docs.github.com/en/actions/sharing-automations/reusing-workflows#using-inputs-and-secrets-in-a-reusable-workflow

Done in commit e91040b. The inputs let us create a daily artifact. I'm using that artifact for comparisons.

mpaulucci · 2025-01-30T17:40:59Z

.github/workflows/ci_levm.yaml

@@ -58,6 +58,49 @@ jobs:
              echo "Percentage is not 100%."
              exit 1
          fi
+
+  hive-report-creation:
+    uses: ./.github/workflows/common_hive_reports.yaml


since these are ran daily, can't we take the results from the last (levm) daily run and compare it to the current run?

I thinks it's done in commit e91040b. We would have to merge it to fully test it

.github/workflows/daily_reports.yaml

test: ci

2dccc09

fborello-lambda added the levm Lambda EVM implementation label Jan 29, 2025

fborello-lambda added 8 commits January 29, 2025 17:04

refactor: ci

f288774

ci: continue-on-error

008dea7

test: ci

e9c0683

feat: update comment only if it exists

fe4071b

undo test change

f088287

test with comment

d6304f3

test

5567f74

create or update on failure

84ad4ca

lambdaclass deleted a comment from github-actions bot Jan 29, 2025

fborello-lambda added 8 commits January 29, 2025 17:49

test

c04e4f1

test

600d7a6

test

10a6f99

undo test change

30e3718

feat: hive diff msg only on change

4abf742

feat: hive test in ci_levm

bc72374

Merge branch 'main' into levm/ci_silence_ef_tests_msg

1d35616

test

e81dc4e

fborello-lambda self-assigned this Jan 30, 2025

fborello-lambda added 2 commits January 30, 2025 10:13

fix: RUST version

9429b11

fix: report Check

b7890a0

fborello-lambda force-pushed the levm/ci_silence_ef_tests_msg branch from fc19d04 to b7890a0 Compare January 30, 2025 13:49

undo test msg

3a186fe

fborello-lambda marked this pull request as ready for review January 30, 2025 14:52

fborello-lambda requested a review from a team as a code owner January 30, 2025 14:52

mpaulucci reviewed Jan 30, 2025

View reviewed changes

.github/workflows/common_hive_reports.yaml Outdated Show resolved Hide resolved

mpaulucci reviewed Jan 30, 2025

View reviewed changes

.github/workflows/daily_reports.yaml Show resolved Hide resolved

fborello-lambda added 4 commits January 30, 2025 15:11

pr_comments: single Integration Test

5f3a652

pr_comments: common_hive_reports.yaml -> common/hive_reports.yaml

054089e

pr_comments: include inputs, compare against main

e91040b

fix and test

960302f

fborello-lambda force-pushed the levm/ci_silence_ef_tests_msg branch from fdada83 to 960302f Compare January 30, 2025 19:39

test ci

9fc4ee4

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ci(levm): silence messages #1837

ci(levm): silence messages #1837

fborello-lambda commented Jan 29, 2025 •

edited

Loading

github-actions bot commented Jan 29, 2025 •

edited

Loading

github-actions bot commented Jan 29, 2025 •

edited

Loading

github-actions bot commented Jan 29, 2025 •

edited

Loading

mpaulucci Jan 30, 2025

fborello-lambda Jan 30, 2025

mpaulucci Jan 30, 2025

fborello-lambda Jan 30, 2025

fborello-lambda Jan 30, 2025

mpaulucci Jan 30, 2025

fborello-lambda Jan 30, 2025

fborello-lambda Jan 30, 2025

mpaulucci Jan 30, 2025

fborello-lambda Jan 30, 2025

mpaulucci Jan 30, 2025

fborello-lambda Jan 30, 2025 •

edited

Loading

ci(levm): silence messages #1837

Are you sure you want to change the base?

ci(levm): silence messages #1837

Conversation

fborello-lambda commented Jan 29, 2025 • edited Loading

github-actions bot commented Jan 29, 2025 • edited Loading

github-actions bot commented Jan 29, 2025 • edited Loading

Benchmark Results Comparison

PR Results

Benchmark Results: Factorial

Benchmark Results: Factorial - Recursive

Benchmark Results: Fibonacci

Benchmark Results: ManyHashes

Benchmark Results: ERC20 - Transfer

Benchmark Results: ERC20 - Mint

Benchmark Results: ERC20 - Approval

Main Results

Benchmark Results: Factorial

Benchmark Results: Factorial - Recursive

Benchmark Results: Fibonacci

Benchmark Results: ManyHashes

Benchmark Results: ERC20 - Transfer

Benchmark Results: ERC20 - Mint

Benchmark Results: ERC20 - Approval

github-actions bot commented Jan 29, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fborello-lambda Jan 30, 2025 • edited Loading

Choose a reason for hiding this comment

fborello-lambda commented Jan 29, 2025 •

edited

Loading

github-actions bot commented Jan 29, 2025 •

edited

Loading

github-actions bot commented Jan 29, 2025 •

edited

Loading

github-actions bot commented Jan 29, 2025 •

edited

Loading

fborello-lambda Jan 30, 2025 •

edited

Loading