Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Build, test, and promote automation for 3P Packages #220

Merged

Conversation

amzn-changml
Copy link
Contributor

This implements a modified version of the 3P Package automation RFC: https://github.com/o3de/sig-build/blob/main/rfcs/rfc-bld-20230302-1-3p-development.md

An overview of the workflow (will be captured in a README):

Package create/modify

  1. User makes a modification to package_build_list_host_*.json in a branch/fork and issues a PR
  2. build-packages.yaml will trigger on a PR targeting main with a change to package_build_list_host_*.json
  3. A script will diff the change and look for any bump in version
  4. If (3) is successful, run parallel builds against the change for Mac, Windows, Linux (Ubuntu), and Linux-aarch64 (Ubuntu) using 3p-package-scripts/o3de_package_scripts/build_package.py --search_path 3p-package-source <name of package>
  5. If (4) is successful, validate change (virus scan and check SHA256)
  6. Push the built package to Github Artifacts, accessible from the job status screen
  7. User merges the PR once approvals and status checks are completed

Package promotion

  1. Once the PR is merged, promote-packages.yaml will trigger, if package_build_list_host_*.json is modified
  2. An automation will pull the last successful build from the package create/modify phase and download the Github Artifacts
  3. The 3P packages downloaded from (2) is pushed to a dev S3 bucket, accessible from AR. This will not start until a maintainer approves this step through a Github dialog box
  4. The SHA256SUMS file is pulled out for each package
  5. The package name and SHA256 is replaced in the O3DE repo path cmake/3rdParty/Platform/<platform>/BuiltInPackages_<platform>.cmake within a PR branch
  6. A automated PR is created with the changes from (4)
  7. Once approval of the PR is completed, there is a second dialog box will be available for a maintainer to approve pushing the packages from the dev S3 to the prod S3
  8. Once the push to the prod S3 is complete, a maintainer can merge the PR to O3DE

Testing done
This was developed in tested in my fork against a real 3P promotion PR for dxc (#211). An example of the workflows below:

Build automation: https://github.com/amzn-changml/3p-package-source/actions/runs/6241586122 (note the downloadable artifacts at the bottom of the screen)
Promotion automation: https://github.com/amzn-changml/3p-package-source/actions/runs/6281273986

Automated PR for O3DE: amzn-changml/o3de#34

Container creation automation (to allow for aarch64 builds with qemu): https://github.com/amzn-changml/3p-package-source/actions/runs/6215325592

Signed-off-by: Mike Chang <changml@amazon.com>
@amzn-changml amzn-changml requested review from a team and nick-l-o3de September 25, 2023 17:56
.github/workflows/build-container.yaml Outdated Show resolved Hide resolved
.github/workflows/build-package.yaml Outdated Show resolved Hide resolved
.github/workflows/promote-packages.yaml Outdated Show resolved Hide resolved
.github/workflows/build-package.yaml Outdated Show resolved Hide resolved
amzn-changml and others added 2 commits September 25, 2023 11:52
Co-authored-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
Signed-off-by: Mike Chang <changml@amazon.com>
Signed-off-by: Mike Chang <changml@amazon.com>
name: Build 3P Packages

on:
pull_request:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we add types: [opened, reopened] to limit when we trigger this workflow

Also do we want to enable running this on PR from a fork or just supported branches?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should rebuild on changes, not just when the PR is opened, but good callout to limit builds. I'll see if we can put the build portion behind the github dialog for environments. I'll also see about using the workflow_dispatch trigger so that someone in a fork can trigger manually.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, this won't retrigger unless package_build_list_host_*.json is changed. If someone is making a further change to their PR that doesn't involve editing package_build_list_host_*.json again, but requires rebuild, they can just do a re-run of the build job, since it'll pull from latest

.github/workflows/build-package.yaml Outdated Show resolved Hide resolved
Signed-off-by: Mike Chang <changml@amazon.com>
Signed-off-by: Mike Chang <changml@amazon.com>
@amzn-changml amzn-changml merged commit 81deb51 into o3de:main Sep 26, 2023
1 check passed
@amzn-changml amzn-changml deleted the devops/automated-build-deploy branch September 26, 2023 20:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants