Skip to content

Commit ca8cd7a

Browse files
TerrenceMcGuinness-NOAAtmcguinness
andauthored
Auto provisioning of PW clusters from GitHub CI added (#3051)
# Description This update to the GitHub dispatched CI pipeline to execute the self-hosted GitHub Runner on Parallel Works now adds the feature that starts up the virtual compute cluster automatically. We now have a complete end-to-end automated process for running CI tests in Parallel Works. Next steps would be tear-down and adding more test to see if it scales. It also has the update for getting a PR to load up when its originating from a forked repo. # Type of change - [ ] Bug fix (fixes something broken) - [x] New feature (adds functionality) - [ ] Maintenance (code refactor, clean-up, new CI test, etc.) # Change characteristics <!-- Choose YES or NO from each of the following and delete the other --> - Is this a breaking change (a change in existing functionality)? NO - Does this change require a documentation update? YES - Does this change require an update to any of the following submodules? NO (If YES, please add a link to any PRs that are pending.) - [ ] EMC verif-global <!-- NOAA-EMC/EMC_verif-global#1234 --> - [ ] GDAS <!-- NOAA-EMC/GDASApp#1234 --> - [ ] GFS-utils <!-- NOAA-EMC/gfs-utils#1234 --> - [ ] GSI <!-- NOAA-EMC/GSI#1234 --> - [ ] GSI-monitor <!-- NOAA-EMC/GSI-Monitor#1234 --> - [ ] GSI-utils <!-- NOAA-EMC/GSI-Utils#1234 --> - [ ] UFS-utils <!-- ufs-community/UFS_UTILS#1234 --> - [ ] UFS-weather-model <!-- ufs-community/ufs-weather-model#1234 --> - [ ] wxflow <!-- NOAA-EMC/wxflow#1234 --> # How has this been tested? The start up aspected has been tested from my forked repo but could not test repos that are forked. The test from forked repos has to be tested once the workflow pipeline in the **develop** branch. # Checklist - [x] Any dependent changes have been merged and published - [x] My code follows the style guidelines of this project - [x] I have performed a self-review of my own code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have documented my code, including function, input, and output descriptions - [x] My changes generate no new warnings - [x] New and existing tests pass with my changes - [x] This change is covered by an existing CI test or a new one has been added - [ ] Any new scripts have been added to the .github/CODEOWNERS file with owners - [ ] I have made corresponding changes to the system documentation if necessary --------- Co-authored-by: tmcguinness <terry.mcguinness@noaa.gov> Co-authored-by: tmcguinness <tmcguinness@129.qarestr.sub-172-16-82.myvzw.com>
1 parent 5a8a5aa commit ca8cd7a

File tree

1 file changed

+35
-1
lines changed

1 file changed

+35
-1
lines changed

.github/workflows/pw_aws_ci.yaml

Lines changed: 35 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -31,24 +31,57 @@ env:
3131
MACHINE_ID: noaacloud
3232

3333
jobs:
34+
35+
run-start-clusters:
36+
runs-on: ubuntu-latest
37+
env:
38+
PW_PLATFORM_HOST: noaa.parallel.works
39+
steps:
40+
- name: Checkout pw-cluster-automation repository
41+
uses: actions/checkout@v4
42+
with:
43+
repository: TerrenceMcGuinness-NOAA/pw-cluster-automation
44+
path: pw-cluster-automation
45+
ref: pw_cluster_noaa
46+
47+
- name: Run startClusters
48+
run: |
49+
mkdir -p ~/.ssh
50+
echo "${{ secrets.ID_RSA_AWS }}" > ~/.ssh/id_rsa
51+
echo "${{ secrets.PW_API_KEY }}" > ~/.ssh/pw_api.key
52+
chmod 700 ~/.ssh
53+
chmod 600 ~/.ssh/id_rsa
54+
chmod 600 ~/.ssh/pw_api.key
55+
if [ "${{ github.event.inputs.os }}" == "rocky" ]; then
56+
clustername="globalworkflowciplatformrocky8"
57+
elif [ "${{ github.event.inputs.os }}" == "centos" ]; then
58+
clustername="awsemctmcgc7i48xlargeciplatform"
59+
fi
60+
python3 pw-cluster-automation/startClusters.py $clustername
61+
3462
fetch-branch:
63+
needs: run-start-clusters
3564
runs-on: ubuntu-latest
3665
env:
3766
GH_TOKEN: ${{ secrets.GITHUBTOKEN }}
3867
outputs:
3968
branch: ${{ steps.get-branch.outputs.branch }}
69+
repo: ${{ steps.get-branch.outputs.repo }}
4070
steps:
41-
- name: Fetch branch name for PR
71+
- name: Fetch branch name and repo for PR
4272
id: get-branch
4373
run: |
4474
pr_number=${{ github.event.inputs.pr_number }}
4575
repo=${{ github.repository }}
4676
if [ "$pr_number" -eq "0" ]; then
4777
branch=${{ github.event.inputs.ref }}
78+
repo_url="https://github.com/${{ github.repository_owner }}/${{ github.repository }}.git"
4879
else
4980
branch=$(gh pr view $pr_number --repo $repo --json headRefName --jq '.headRefName')
81+
repo_url=$(gh pr view $pr_number --repo $repo --json headRepository --jq '.headRepository.url')
5082
fi
5183
echo "::set-output name=branch::$branch"
84+
echo "::set-output name=repo::$repo_url"
5285
5386
checkout:
5487
needs: fetch-branch
@@ -64,6 +97,7 @@ jobs:
6497
with:
6598
path: ${{ github.run_id }}/HOMEgfs
6699
submodules: 'recursive'
100+
repository: ${{ needs.fetch-branch.outputs.repo }}
67101
ref: ${{ needs.fetch-branch.outputs.branch }}
68102

69103
build-link:

0 commit comments

Comments
 (0)