build: Add release pipeline

[bgins]

Review Type Requested (choose one):

• Glance - superficial check (from domain experts)

Summary

This pull request adds a release workflow that:

• Automatically generates release pull requests and merges them immediately
• Automatically creates releases when the release pull request it merged
• Extends generated release notes with custom notes
• Builds binaries and uploads them to the release

It also makes a few associated changes:

• Update testnet_deploy_services.yml to deploy only on releases
• Update release_docker.yml to publish only on releases. Removed. We also want to publish on push to main.
• Update resource provider updater script to expect -gpu suffix in release name (and rename script to lilypad-updater-gpu.sh)
• Remove CirclCI build and existing GH actions build and release workflows
• Naming clean up to use Go conventions (https://go.dev/doc/effective_go#mixed-caps)

Task/Issue reference

Closes: #91, closes #168

Details

Goals

Our goals for the release pipeline:

• Release features and fixes on merge to main
• Support trunk-based development
• Automate everything as much as possible

release-please

This PR uses release-please to generate release pull requests and create releases when new features or fixes are introduced. release-please checks for conventional commit prefixes to determine a semantic version and to decide when to create a release. From their documentation:

The most important prefixes you should have in mind are:

• fix: which represents bug fixes, and correlates to a SemVer patch.
• feat: which represents a new feature, and correlates to a SemVer minor.
• feat!:, or fix!:, refactor!:, etc., which represent a breaking change (indicated by the !) and will result in a SemVer major.

release-please does not create release PRs or releases for tasks without user facing (prefixes like chore:, build:, etc).

Automate it more please

A typical release-please setup creates a release pull request that a human approves and merges. Once the PR is merged, release-please runs a second time and creates a release when it observes its release commit.

We take this a step further by automatically merging the release pull request. After we merge, release-please runs a second time to create the release.

Here is a diagram showing an example flow on commits coming into main and how the releases workflow responds to each. The diagram assumes we have started at version 1.0.0:

We have created a lilypad-releases bot (implemented as a GitHub app) to run release-please actions. The custom bot is needed to trigger further actions, such as running the workflow a second time on merge.

In addition, our branch protection rules for main require one PR approval and will require passing tests in the near future. The lilypad-releases bot cannot approve it's own pull request, so we have the default github-actions bot step in for approval.

Customization

release-please automatically maintains a CHANGELOG, generates pull request text, and generates release notes.

We can customize the pull request text by updating release-please-config.json.

We extend the generated release notes with an extend-notes job. This job reads a release_notes.md file and appends it to the generated release notes. We can customize the release notes by updating this file, but note that the changes will persist in the next release unless they are changed again.

How to test this code?

This pull request is challenging to test in this repository because we need to merge it to see it in action.

I have created a Lilypad-Tech/test-gh-action-releases repo to demonstrate the release workflow with a few test PRs ready to approve and merge: https://github.com/Lilypad-Tech/test-gh-action-releases/pulls

Anything else?

Our internal version to capture the design can be found here: https://www.notion.so/lilypadnetwork/Release-Pipeline-70ecc09bed8b406ba33e7e1e4c4e93b3

[AquiGorka]

NOTES=$(cat << EOF
          $BODY
          $EXTENSION
          EOF
)

The spacing threw me off, thought there was a missing parenthesis

[bgins]

Yeah, admittedly this is an esoteric technique for adding the newlines. But it was the only one that worked out for me implementing it. 😄

[AquiGorka]

Nice! 😍

[AquiGorka]

Are we not doing linux non gpu? Those would be for people running the binary in docker? (assuming most docker images use some flavor of linux)

[bgins]

We do a CPU-only build for every OS and architecture. The gpu flag indicates whether we should also do a GPU build.

[AquiGorka]

Thank you for clarifying, I misread the matrix

[AquiGorka]

We take this a step further by automatically merging the release pull request. After we merge, release-please runs a second time to create the release.

😍

Thank you @bgins 🎉

[walkah]

One small Q, but looking forward to having this in place! Great work, @bgins 👏 🚀

[walkah]

We don't want a latest docker image that tracks main (for testing, etc)?

[bgins]

Ah yeah, good point. Two questions about it.

Are there changes that are not user facing that should trigger a new Docker image? With the current setup, we will create a new image for every feat:, fix:, and breaking change. Are there cases where we would want them for chore:, build:, etc?

Second, does ghcr have a mechanism for listing images by git tag? They don't list our existing tags ((https://github.com/Lilypad-Tech/lilypad/pkgs/container/resource-provider/versions), but maybe it's because of the way we have been formatting them. The concern here is whether or not resource providers can find the latest released version vs the latest published version.

[walkah]

Are there changes that are not user facing that should trigger a new Docker image? With the current setup, we will create a new image for every feat:, fix:, and breaking change. Are there cases where we would want them for chore:, build:, etc?

Maybe not? I think I don't know the answer yet.

Second, does ghcr have a mechanism for listing images by git tag? They don't list our existing tags ((https://github.com/Lilypad-Tech/lilypad/pkgs/container/resource-provider/versions), but maybe it's because of the way we have been formatting them. The concern here is whether or not resource providers can find the latest released version vs the latest published version.

It will publish on git tags - I think we just haven't done one since the docker build step was added.

[bgins]

Alright yeah, if the tagged versions will be distinct on ghcr, let's publish images on push to main and tags. Will drop the change.

Can imagine a scenario where we are updating the Dockerfile, it's a build: task not a fix/feature, and we want the team to have easy access to an image.