Skip to content
This repository has been archived by the owner on Sep 23, 2024. It is now read-only.

feature: e2e regression testing tool #119

Merged
merged 46 commits into from
Apr 29, 2024
Merged
Changes from 36 commits
Commits
Show all changes
46 commits
Select commit Hold shift + click to select a range
feeee91
feature: e2e regression testing tool
rebelArtists Apr 10, 2024
8e304f7
debug
rebelArtists Apr 10, 2024
238594e
fix: verifiedBatchNumber query
rebelArtists Apr 10, 2024
221aa95
debug verifiedBatch query
rebelArtists Apr 10, 2024
18cb797
naming convention
rebelArtists Apr 10, 2024
66b5d96
formatting and pr cleanup
rebelArtists Apr 10, 2024
3fd15cd
flakey unittests
rebelArtists Apr 10, 2024
0266c41
trying to circumvent flakey unittests
rebelArtists Apr 10, 2024
60568ac
pr cleanup
rebelArtists Apr 10, 2024
845b110
fix: kurtosis install
rebelArtists Apr 10, 2024
de9906b
fix: typo
rebelArtists Apr 10, 2024
937a593
auto run for commits to main
rebelArtists Apr 10, 2024
536b3fa
fix: auto regression runs for main branch
rebelArtists Apr 10, 2024
a0d8faf
debug: short sha
rebelArtists Apr 10, 2024
750f703
fix: commit sha
rebelArtists Apr 10, 2024
1bdd043
fix: commit sha again
rebelArtists Apr 10, 2024
e7db067
fix: sha
rebelArtists Apr 10, 2024
87905f6
debug
rebelArtists Apr 10, 2024
4464879
debug
rebelArtists Apr 10, 2024
ad869e3
debuggg
rebelArtists Apr 10, 2024
3dfeb12
debugggg
rebelArtists Apr 10, 2024
a416c09
debug
rebelArtists Apr 10, 2024
8c97c8f
debugin
rebelArtists Apr 10, 2024
26f5c24
debuginn
rebelArtists Apr 10, 2024
a4d88a3
debuggin
rebelArtists Apr 10, 2024
eb09d95
debuggin still
rebelArtists Apr 10, 2024
92ad853
debuggg
rebelArtists Apr 10, 2024
deeec65
cleanup
rebelArtists Apr 10, 2024
997a0d3
fix: verifiedBatch polling
rebelArtists Apr 10, 2024
85a98f1
consolidate files
rebelArtists Apr 16, 2024
5a06766
fix: build
rebelArtists Apr 16, 2024
9fdd47b
debug
rebelArtists Apr 16, 2024
56ac08e
build img
rebelArtists Apr 16, 2024
417d2f4
final checks
rebelArtists Apr 16, 2024
80924f7
final checkss
rebelArtists Apr 16, 2024
835a704
cmon
rebelArtists Apr 16, 2024
62da1f6
touchup per pr feedback
rebelArtists Apr 26, 2024
aefa1fd
fix: clone
rebelArtists Apr 26, 2024
1bbd617
debug
rebelArtists Apr 26, 2024
47cdaeb
debuggin
rebelArtists Apr 26, 2024
d4b4f75
fix
rebelArtists Apr 26, 2024
1ad0a5f
cleanup
rebelArtists Apr 26, 2024
098fbd5
debug
rebelArtists Apr 26, 2024
fa3b13e
debug
rebelArtists Apr 26, 2024
2c69a1b
debug
rebelArtists Apr 26, 2024
861e7b2
demo
rebelArtists Apr 26, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
76 changes: 76 additions & 0 deletions .github/workflows/regression-test-auto.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,76 @@
name: Regression Tests - Agglayer
'on':
pull_request:
types: [opened, synchronize] # Trigger on new pull requests and when existing ones are synchronized
branches:
- main
inputs:
bake_time:
description: bake time (minutes)
required: false
default: 20
jobs:
deploy_devnet:
runs-on: ubuntu-latest
steps:
- name: Set up Docker
uses: docker/setup-buildx-action@v1
- name: Clone and build agglayer image locally
run: |
git clone https://github.com/0xPolygon/agglayer.git
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this really needed if the current repo is already agglayer?

Copy link

@leovct leovct Apr 24, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this really needed if the current repo is already agglayer?

good call! and I think we should build the Docker image using Dockerfile.release.

https://github.com/0xPolygon/kurtosis-cdk/blob/main/.github/workflows/regression-tests.yml#L52-L56

Screenshot 2024-04-24 at 10 43 22

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed, good call

cd agglayer
if [ "$GITHUB_EVENT_NAME" == "pull_request" ]; then
GITHUB_SHA=$(cat $GITHUB_EVENT_PATH | jq -r .pull_request.head.sha)
fi
git checkout $GITHUB_SHA
echo "Containerizing and testing commit: $GITHUB_SHA"
docker compose -f docker/docker-compose.yaml build --no-cache agglayer
- name: Install Foundry
uses: foundry-rs/foundry-toolchain@v1
- name: Clone internal kurtosis-cdk repo
run: |
git clone https://github.com/0xPolygon/kurtosis-cdk.git
cd kurtosis-cdk
git checkout dan/jit_containers
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are we going to merge this?

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yup it's not needed anymore. We would just need to install yq and replace the zkevm_agglayer_image value by the name of the docker image (e.g. agglayer:local).

For reference: https://github.com/0xPolygon/kurtosis-cdk/blob/main/.github/workflows/regression-tests.yml#L102-L115

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed!

- name: Install kurtosis
run: |
echo "deb [trusted=yes] https://apt.fury.io/kurtosis-tech/ /" | sudo tee /etc/apt/sources.list.d/kurtosis.list
sudo apt update
sudo apt install kurtosis-cli
rebelArtists marked this conversation as resolved.
Show resolved Hide resolved
kurtosis analytics disable
- name: Deploy CDK devnet on local github runner
run: |
cd kurtosis-cdk
kurtosis run --enclave cdk-v1 --args-file params.yml .
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we use a local params file instead of relying on the version matrix in the kurtosis repo?

Copy link

@leovct leovct Apr 24, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This file defines the configuration of the entire CDK stack, not just the agglayer. In the context of e2e tests, I think it's interesting to use the kurtosis-cdk configuration file. It allows you to test the agglayer in contact with the latest versions of the other components (because we maintain the tool and we're iterating fast at the moment) and, what's more, you have one less file to maintain. But up to you :)

- name: Monitor and report any potential regressions to CI logs
run: |
bake_time=30 # minutes
end_minute=$((10#$(date +'%M') + bake_time))

export ETH_RPC_URL="$(kurtosis port print cdk-v1 zkevm-node-rpc-001 http-rpc)"
INITIAL_STATUS=$(cast rpc zkevm_verifiedBatchNumber 2>/dev/null)
incremented=false

while [ $(date +'%M') -lt $end_minute ]; do
# Attempt to connect to the service
if STATUS=$(cast rpc zkevm_verifiedBatchNumber 2>/dev/null); then
echo "ZKEVM_VERIFIED_BATCH_NUMBER: $STATUS"

# Check if STATUS has incremented
if [ "$STATUS" != "$INITIAL_STATUS" ]; then
incremented=true
echo "ZKEVM_VERIFIED_BATCH_NUMBER successfully incremented to $STATUS. Exiting..."
exit 0
fi
else
echo "Failed to connect, waiting and retrying..."
sleep 60
continue
fi
sleep 60
done

if ! $incremented; then
echo "ZKEVM_VERIFIED_BATCH_NUMBER did not increment. This may indicate chain experienced a regression. Please investigate."
exit 1
fi
Loading