Skip to content

Commit

Permalink
Workflows: reworked actions and ways to skip eamxx testing jobs
Browse files Browse the repository at this point in the history
* Do not check labels in an action.
* Implement get-labels mini-action
* Add "check" jobs that, for PRs, verify that
  * The PR touches relevant jobs
  * The PR does not have "skip" labels
  • Loading branch information
bartgol committed Nov 6, 2024
1 parent e902e38 commit 3118127
Show file tree
Hide file tree
Showing 9 changed files with 259 additions and 140 deletions.
29 changes: 29 additions & 0 deletions .github/actions/check-pr-paths/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
# Composite action to check if testing for this PR is relevant based on touched paths

This action is meant to be used inside a PR testing workflow, as

```yaml
jobs:
check_paths:
steps:
- name: check_relevance
if: ${{ github.event_name == "pull_request" }}
uses: ./.github/actions/check-pr-paths
with:
paths: |
path1
path2
testing:
needs: check_paths
if: ${{ github.event_name != "pull_request" || needs.check_paths.outputs.touched == 'true'}}
steps:
- name: testing
```

Check failure on line 21 in .github/actions/check-pr-paths/README.md

View workflow job for this annotation

GitHub Actions / linter

Fenced code blocks should be surrounded by blank lines

.github/actions/check-pr-paths/README.md:21 MD031/blanks-around-fences Fenced code blocks should be surrounded by blank lines [Context: "```"] https://github.com/DavidAnson/markdownlint/blob/v0.35.0/doc/md031.md
The input `paths1 is a list of paths that the action uses to see if
the PR changes any relevant file. The action sets an output `touched`,

Check failure on line 23 in .github/actions/check-pr-paths/README.md

View workflow job for this annotation

GitHub Actions / linter

Spaces inside code span elements

.github/actions/check-pr-paths/README.md:23:1 MD038/no-space-in-code Spaces inside code span elements [Context: "...e. The action sets an output `"] https://github.com/DavidAnson/markdownlint/blob/v0.35.0/doc/md038.md
which is `true` if the PR touches any of the listed paths and `false`

Check failure on line 24 in .github/actions/check-pr-paths/README.md

View workflow job for this annotation

GitHub Actions / linter

Spaces inside code span elements

.github/actions/check-pr-paths/README.md:24:1 MD038/no-space-in-code Spaces inside code span elements [Context: "which is `"] https://github.com/DavidAnson/markdownlint/blob/v0.35.0/doc/md038.md
otherwise.

This action allows to skip an entire workflow, but, unlike using a 'paths' filter at the
workflow trigger level, it ensures that the workflow DOES run,
hence avoiding the issue where certain checks are not present on the PR.
46 changes: 46 additions & 0 deletions .github/actions/check-pr-paths/action.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
name: check-pr-paths
description: 'Check that the PR modifies relevant files'
inputs:
paths:
description: 'List of paths that are considered relevant'
required: true
default: ''
token:
description: 'GitHub token for authentication'
required: true
outputs:
touched:
description: 'Output indicating if this testing for PR is relevant'
value: ${{ steps.check_paths.outputs.value }}

# Note: inputs are available as env vars in the shell run steps, convertet to uppercase
runs:
using: "composite"
steps:
- name: check_paths
shell: bash
env:
PR_NUMBER: ${{ github.event.number }}
GITHUB_REPO: ${{ github.repository }}
TOKEN: ${{ inputs.token }}
run: |
# Convert the input paths into an array, and create a regex from it
IFS=$'\n' read -r -a paths <<< "$INPUT_PATHS"
pattern=$(IFS=\|; echo "${paths[*]}")
# Use the GitHub API to get the list of changed files
response=$(curl -s -H "Authorization: token $TOKEN" \
"https://api.github.com/repos/$GITHUB_REPO/pulls/$PR_NUMBER/files")
# Extract filenames using grep and sed
changed_files=$(echo "$response" | grep -o '"filename": *"[^"]*"' | sed 's/"filename": *//; s/"//g')
# Check for matches and echo the matching files
matching_files=$(echo "$changed_files" | grep -E "^($pattern)" || echo "")
if [[ -n "$matching_files" ]]; then
echo "Found relevant files: $matching_files" # Print the matching files
echo "touched=true" >> $GITHUB_OUTPUT
else
echo "No relevant files touched by this PR."
echo "touched=false" >> $GITHUB_OUTPUT
fi
24 changes: 0 additions & 24 deletions .github/actions/check-skip-labels/README.md

This file was deleted.

48 changes: 0 additions & 48 deletions .github/actions/check-skip-labels/action.yml

This file was deleted.

19 changes: 19 additions & 0 deletions .github/actions/get-labels/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
# Composite action to retrieve labels of a PR if event is `pull_request`

This action is meant to be used inside a PR testing workflow, as

```yaml
jobs:
get_labels:
steps:
- name: get_labels
uses: ./.github/actions/get-labels
testing:
needs: get_labels
if: ${{ github.event_name != "pull_request" ||
!contains(needs.get_labels.outputs.labels,"label1" ||
!contains(needs.get_labels.outputs.labels,"label1" }}
```

Check failure on line 16 in .github/actions/get-labels/README.md

View workflow job for this annotation

GitHub Actions / linter

Fenced code blocks should be surrounded by blank lines

.github/actions/get-labels/README.md:16 MD031/blanks-around-fences Fenced code blocks should be surrounded by blank lines [Context: "```"] https://github.com/DavidAnson/markdownlint/blob/v0.35.0/doc/md031.md
The action sets the output `labels`, which is a comma-separated list
containing the PR labels. If the event that triggered the workflow is
not a PR, the output will be an empty string
17 changes: 17 additions & 0 deletions .github/actions/get-labels/action.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
name: get-labels
description: 'Check that the PR modifies does not have skip labels'
outputs:
labels:
description: 'Comma-separated list of labels on the PR or empty string (if not a PR event)'
value: ${{ steps.get_labels.outputs.labels }}

# Note: inputs are available as env vars in the shell run steps, convertet to uppercase
runs:
using: "composite"
steps:
- name: get_labels
shell: sh
env:
PR_LABELS: ${{ toJson(github.event.pull_request.labels.*.name) }} # Pass PR label names as JSON
run: |
echo 'labels=\"$(echo \"$PR_LABELS\" | jq -r "join(\",\")")\"' >> $GITHUB_OUTPUT
83 changes: 52 additions & 31 deletions .github/workflows/eamxx-sa-testing.yml
Original file line number Diff line number Diff line change
Expand Up @@ -5,17 +5,6 @@ on:
pull_request:
branches: [ master ]
types: [opened, synchronize, ready_for_review, reopened]
paths:
- components/eamxx/**
- components/eam/src/physics/rrtmgp/**
- components/eam/src/physics/p3/scream/**
- components/eam/src/physics/cam/**
- .github/workflows/eamxx-standalone-testing.yml
- externals/ekat/**
- externals/scorpio/**
- externals/haero/**
- externals/YAKL/**
- components/eam/src/physics/rrtmgp/external/**

# Manual run is used to bless
workflow_dispatch:
Expand Down Expand Up @@ -48,15 +37,51 @@ env:
submit: ${{ github.event_name == 'schedule' && 'true' || 'false' }} # Submit to cdash only for nightlies

jobs:
pr_relevant:
runs-on: ubuntu-latest # This job can run anywhere
outputs:
value: ${{ steps.check_paths.outputs.touched }}
steps:
- name: check_paths
if: ${{ github.event_name == "pull_request" }}
uses: ./.github/actions/check-pr-relevance
with:
paths: |
components/eamxx
components/eam/src/physics/rrtmgp
components/eam/src/physics/p3/scream
components/eam/src/physics/cam
components/eam/src/physics/rrtmgp/external
externals/ekat
externals/scorpio
externals/haero
externals/YAKL
.github/workflows/eamxx-sa-testing.yml
get_labels:
runs-on: ubuntu-latest
outputs: ${{ steps.get_labels.outputs.labels }}
steps:
- name: get_labels
uses: ./.github/actions/get-labels
gcc-openmp:
needs: pr_relevant, get_labels
if: ${{
(github.event_name == 'pull_request' &&
needs.pr_relevant.outputs.value=='true' &&
!contains(needs.get_labels.outputs.labels,'CI: skip gcc') &&
!contains(needs.get_labels.outputs.labels,'CI: skip openmp') &&
!contains(needs.get_labels.outputs.labels,'CI: skip eamxx-sa') &&
!contains(needs.get_labels.outputs.labels,'CI: skip eamxx-all')) ||
(github.event_name == 'workflow_dispatch' &&
github.event.inputs.job_to_run == 'gcc-openmp' ||
github.event.inputs.job_to_run == 'all') ||
github.event_name == 'schedule'
}}
runs-on: [self-hosted, ghci-snl-cpu, gcc]
strategy:
fail-fast: false
matrix:
build_type: [sp, dbg, fpe, opt]
if: ${{ github.event_name != 'workflow_dispatch' ||
github.event.inputs.job_to_run == 'gcc-openmp' ||
github.event.inputs.job_to_run == 'all' }}
name: gcc-openmp / ${{ matrix.build_type }}
steps:
- name: Check out the repository
Expand All @@ -67,13 +92,6 @@ jobs:
submodules: recursive
- name: Show action trigger
uses: ./.github/actions/show-workflow-trigger
- name: Check for skip labels
if: ${{ github.event_name == 'pull_request' || github.event_name == 'pull_request_review' }}
uses: ./.github/actions/check-skip-labels
with:
skip_labels: 'AT: skip gcc,AT: skip openmp,AT: skip eamxx-sa,AT: skip eamxx-all'
token: ${{ secrets.GITHUB_TOKEN }}
pr_number: ${{ github.event.pull_request.number }}
- name: Set test-all inputs based on event specs
run: |
echo "generate=false" >> $GITHUB_ENV
Expand All @@ -91,14 +109,24 @@ jobs:
submit: ${{ env.submit }}
cmake-configs: Kokkos_ENABLE_OPENMP=ON
gcc-cuda:
needs: pr_relevant, get_labels
if: ${{
(github.event_name == 'pull_request' &&
needs.pr_relevant.outputs.value=='true' &&
!contains(needs.get_labels.outputs.labels,'CI: skip gcc') &&
!contains(needs.get_labels.outputs.labels,'CI: skip cuda') &&
!contains(needs.get_labels.outputs.labels,'CI: skip eamxx-sa') &&
!contains(needs.get_labels.outputs.labels,'CI: skip eamxx-all')) ||
(github.event_name == 'workflow_dispatch' &&
github.event.inputs.job_to_run == 'gcc-cuda' ||
github.event.inputs.job_to_run == 'all') ||
github.event_name == 'schedule'
}}
runs-on: [self-hosted, ghci-snl-cuda, cuda, gcc]
strategy:
fail-fast: false
matrix:
build_type: [sp, dbg, opt]
if: ${{ github.event_name != 'workflow_dispatch' ||
github.event.inputs.job_to_run == 'gcc-cuda' ||
github.event.inputs.job_to_run == 'all' }}
name: gcc-cuda / ${{ matrix.build_type }}
steps:
- name: Check out the repository
Expand All @@ -109,13 +137,6 @@ jobs:
submodules: recursive
- name: Show action trigger
uses: ./.github/actions/show-workflow-trigger
- name: Check for skip labels
if: ${{ github.event_name == 'pull_request' || github.event_name == 'pull_request_review' }}
uses: ./.github/actions/check-skip-labels
with:
skip_labels: 'AT: skip gcc,AT: skip cuda,AT: skip eamxx-sa,AT: skip eamxx-all'
token: ${{ secrets.GITHUB_TOKEN }}
pr_number: ${{ github.event.pull_request.number }}
- name: Set test-all inputs based on event specs
run: |
echo "generate=false" >> $GITHUB_ENV
Expand Down
Loading

0 comments on commit 3118127

Please sign in to comment.