-
Notifications
You must be signed in to change notification settings - Fork 42
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
ci(levm): add comparison between ef tests (#1894)
**Motivation** <!-- Why does this pull request exist? What are its goals? --> Now that we have incorporated new tests, it would be nice if we could compare the EF-tests result of the current PR against main. **Description** Incorporate: - `compare-ef-tests` job that compares the results of the PR and the main branch. - This will post a comment in the PR that will only show the table when there's a difference - Update the Summary text (Previously it only showed until `Frontier` forks, now it will show all forks).
- Loading branch information
1 parent
5eddb99
commit 2e91897
Showing
4 changed files
with
209 additions
and
6 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,41 @@ | ||
| #!/bin/bash | ||
|
|
||
| # $1 Main branch tests results | ||
| # $2 PR branch tests results | ||
| main_results=$(cat "$1") | ||
| IFS=$'\n' read -rd '' -a main_results <<<"${main_results}" | ||
|
|
||
|
|
||
| pr_results=$(cat "$2") | ||
| IFS=$'\n' read -rd '' -a pr_results <<<"${pr_results}" | ||
|
|
||
|
|
||
| echo "# EF Tests Comparison" | ||
| echo "|Test Name | MAIN | PR | DIFF | " | ||
| echo "|----------|----------|----|------|" | ||
|
|
||
| num=0 | ||
| for i in "${main_results[@]}" | ||
| do | ||
| name_main=$(echo "$i" | awk -F " " '{print $1}') | ||
| result_main=$(echo "$i" | awk -F " " '{print $2}') | ||
| result_main=${result_main%(*} | ||
|
|
||
| name_pr=$(echo "${pr_results[num]}" | awk -F " " '{print $1}') | ||
| result_pr=$(echo "${pr_results[num]}" | awk -F " " '{print $2}') | ||
| result_pr=${result_pr%(*} | ||
|
|
||
| emoji="" | ||
| if (( $(echo "$result_main > $result_pr" |bc -l) )); then | ||
| emoji="⬇️️" | ||
| elif (( $(echo "$result_main < $result_pr" |bc -l) )); then | ||
| emoji="⬆️" | ||
| else | ||
| emoji="➖️" | ||
| fi | ||
|
|
||
| echo "|$name_main|$result_main|$result_pr| $emoji |" | ||
|
|
||
| num=$((num + 1)) | ||
|
|
||
| done |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,18 @@ | ||
| #!/bin/bash | ||
|
|
||
| # To determine where the test summary ends, we use new lines. | ||
|
|
||
| # Remove everything until the line that says "Summary: " | ||
| resulting_text=$(awk '/Summary: /, 0' $1) | ||
|
|
||
| empty_lines=$(echo "${resulting_text}" | awk '/^$/{print NR}') | ||
| empty_lines=($empty_lines) | ||
|
|
||
| resulting_text=$(echo "${resulting_text}" | sed -e "${empty_lines[0]}d") | ||
|
|
||
| # We substract one because we deleted one before. This correction | ||
| # shouldn't be needed if all lines are deleted as once | ||
| empty_lines[1]=$((empty_lines[1] - 1)) | ||
|
|
||
| resulting_text=$(echo "${resulting_text}" | sed -e "${empty_lines[1]},\$d") | ||
| echo "${resulting_text}" |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters