Assorted release maintenance

.github/workflows/release.yaml

@@ -37,16 +37,16 @@ jobs:
             exit 1
           fi
 
-      - name: setup go ${{ env.GO_VERSION }}
-        uses: actions/setup-go@v5
-        with:
-          go-version: '${{ env.GO_VERSION }}'
-
       - name: checkout release tag
         run: git checkout ${{ env.RELEASE_VERSION }}
 
+      - name: build release helper docker image
+        run: |
+          docker build -t taproot-assets-release-helper \
+            -f make/builder.Dockerfile make/
+
       - name: build release for all architectures
-        run: make release tag=${{ env.RELEASE_VERSION }}
+        run: make docker-release tag=${{ env.RELEASE_VERSION }}
 
       - name: Create Release
         uses: lightninglabs/gh-actions/action-gh-release@2024.07.24.00

Dockerfile

@@ -39,6 +39,14 @@ RUN apk --no-cache add \
 COPY --from=builder /go/bin/tapcli /bin/
 COPY --from=builder /go/bin/tapd /bin/
 
+# Copy the verification script and keys for signature verification.
+COPY --from=builder \
+  /go/src/github.com/lightninglabs/taproot-assets/scripts/verify-install.sh \
+  /verify-install.sh
+COPY --from=builder \
+  /go/src/github.com/lightninglabs/taproot-assets/scripts/keys \
+  /keys
+
 # Store the SHA256 hash of the binaries that were just produced for later
 # verification.
 RUN sha256sum /bin/tapd /bin/tapcli > /shasums.txt \

docs/release.md

@@ -1,125 +1,112 @@
 # `taproot-assets`'s Reproducible Build System
 
-This package contains the build script that the `taproot-assets` project uses in
-order to build binaries for each new release. As of `go1.13`, with some new
-build flags, binaries are now reproducible, allowing developers to build the
-binary on distinct machines, and end up with a byte-for-byte identical binary.
-However, this wasn't _fully_ solved in `go1.13`, as the build system still
-includes the directory the binary is built into the binary itself. As a result,
-our scripts utilize a work around needed until `go1.13.2`.  
+Our release artifacts are designed to be reproducible, meaning, for any
+release version:
 
-## Building a New Release
+1) Release binaries produced for any target architecture correspond
+   byte-for-byte, independent of what host machine or architecture was
+   used to perform the build. For example: if a darwin-arm64 machine and
+   a linux-amd64 machine each build release binaries for windows-amd64,
+   then those produced binaries will match exactly.
 
-### MacOS
+2) These binaries, as well as the source and vendored dependencies
+   used to build them, are packaged in archives that can be reproduced
+   exactly.
 
-The first requirement is to have [`docker`](https://www.docker.com/)
-installed locally and running. The second requirement is to have `make`
-installed. Everything else (including `golang`) is included in the release
-helper image.
+In each case, we confirm an "exact match" by comparing SHA256 digests.
 
-To build a release, run the following commands:
+## Building a Release
+
+To build a release, ensure `make` and `git` are installed, and that
+[`Docker`](https://www.docker.com/) is installed locally and running.
+Then:
 
 ```shell
-$  git clone https://github.com/lightninglabs/taproot-assets.git
-$  cd taproot-assets
-$  git checkout <TAG> # <TAG> is the name of the next release/tag
-$  make docker-release tag=<TAG>
+$ git clone https://github.com/lightninglabs/taproot-assets.git
+$ cd taproot-assets
+$ git checkout "$TAG"
+$ make docker-release tag="$TAG"
 ```
 
-Where `<TAG>` is the name of the next release of `taproot-assets`.
+Where `$TAG` is the name of the desired release of `taproot-assets`.
 
-### Linux/Windows (WSL)
+Note that you can also run `make release` to avoid the dependency on
+Docker and any base images, but, for verification purposes, you must
+ensure to use the same Go toolchain used by the Docker image in order to
+produce release binaries consistent with it.
 
-No prior set up is needed on Linux or macOS is required in order to build the
-release binaries. However, on Windows, the only way to build the release
-binaries at the moment is by using the Windows Subsystem Linux. One can build
-the release binaries following these steps:
+## Verifying a Release
 
-```shell
-$  git clone https://github.com/lightninglabs/taproot-assets.git
-$  cd taproot-assets
-$  git checkout <TAG> # <TAG> is the name of the next release/tag
-$  make release tag=<TAG>
-```
+To manually verify a release, ensure that `gpg`/`gpg2`, `shasum`, and
+`tar`/`unzip` are installed locally, and then proceed with the following
+steps:
 
-This will then create a directory of the form `taproot-assets-<TAG>` containing
-archives of the release binaries for each supported operating system and
-architecture, and a manifest file containing the hash of each archive.
+1. Download the release manifest (`manifest-$TAG.txt`), as
+   well as any desired detached signatures that have been made
+   for it (typically named `manifest-$SIGNER-$TAG.sig`). These
+   are typically available from the Taproot Assets
+   [Releases page][ghrelease] on GitHub, under the 'Assets' header for
+   any given release.
 
-## Verifying a Release
+   Note that the tag itself can be verified via:
+
+   `git verify-tag "$TAG"`
+
+2. Verify the detached signature of the manifest file with:
+
+   `gpg --verify "manifest-$SIGNER-$TAG.sig" "manifest-$TAG.txt"`
 
-With `go1.13`, it's now possible for third parties to verify release binaries.
-Before this version of `go`, one had to trust the release manager(s) to build the
-proper binary. With this new system, third parties can now _independently_ run
-the release process, and verify that all the hashes of the release binaries
-match exactly that of the release binaries produced by said third parties.
-
-To verify a release, one must obtain the following tools (many of these come
-installed by default in most Unix systems): `gpg`/`gpg2`, `shashum`, and
-`tar`/`unzip`.
-
-Once done, verifiers can proceed with the following steps:
-
-1. Acquire the archive containing the release binaries for one's specific
-   operating system and architecture, and the manifest file along with its
-   signature.
-2. Verify the signature of the manifest file with `gpg --verify
-   manifest-<TAG>.txt.sig`. This will require obtaining the PGP keys which
-   signed the manifest file, which are included in the release notes.
-3. Recompute the `SHA256` hash of the archive with `shasum -a 256 <filename>`,
-   locate the corresponding one in the manifest file, and ensure they match
-   __exactly__.
-
-At this point, verifiers can use the release binaries acquired if they trust
-the integrity of the release manager(s). Otherwise, one can proceed with the
-guide to verify the release binaries were built properly by obtaining `shasum`
-and `go` (matching the same version used in the release):
-
-4. Extract the release binaries contained within the archive, compute their
-   hashes as done above, and note them down.
-5. Ensure `go` is installed, matching the same version as noted in the release
-   notes. 
-6. Obtain a copy of `taproot-assets`'s source code with `git clone
-   https://github.com/lightninglabs/taproot-assets` and checkout the source code of the
-   release with `git checkout <TAG>`.
-7. Proceed to verify the tag with `git verify-tag <TAG>` and compile the
-   binaries from source for the intended operating system and architecture with
-   `make release sys=OS-ARCH tag=<TAG>`.
-8. Extract the archive found in the `taproot-assets-<TAG>` directory created by
-   the release script and recompute the `SHA256` hash of the release binaries
-   (`tapd` and `tapcli`) with `shasum -a 256 <filename>`. These should match
-   __exactly__ as the ones noted above.
-
-## Verifying Docker Images
-
-To verify the `tapd` and `tapcli` binaries inside the
-[official provided docker images](https://hub.docker.com/r/lightninglabs/taproot-assets)
-against the signed, reproducible release binaries, there is a verification
-script in the image that can be called (before starting the container for
-example):
+   PGP public keys of Taproot Assets developers can be found in the
+   scripts/keys subdirectory of the repository root.
+
+3. Procure the other release artifacts, either by building the release from
+   source as [described above](#building-a-release), or by downloading
+   the desired archive(s) containing the release binaries from the
+   Taproot Assets [Releases page][ghrelease] on GitHub.
+
+4. Recompute the SHA256 hash of each artifact with e.g.  `shasum -a 256
+   <filename>`, locate the corresponding digest in the manifest file,
+   and ensure they match exactly.
+
+## Verifying Release Binaries in Official Docker Images
+
+To verify the `tapd` and `tapcli` binaries inside the [official provided
+Docker images](https://hub.docker.com/r/lightninglabs/taproot-assets)
+against the signed, reproducible release binaries, there is a
+verification script in the image that can be called (before starting the
+container for example):
 
 ```shell
-$  docker run --rm --entrypoint="" lightninglabs/taproot-assets:v0.3.0 /verify-install.sh v0.3.0
-$  OK=$?
-$  if [ "$OK" -ne "0" ]; then echo "Verification failed!"; exit 1; done
-$  docker run lightninglabs/taproot-assets [command-line options]
+$ docker run --rm --entrypoint="" \
+    lightninglabs/taproot-assets:"$TAG" /verify-install.sh "$TAG"
+$ OK=$?
+$ if [ "$OK" -ne "0" ]; then echo "Verification failed!"; exit 1; done
+$ docker run lightninglabs/taproot-assets [command-line options]
 ```
 
-# Signing an Existing Manifest File
+Note that Docker images published for versions v0.7.0 and earlier don't
+support this script.
 
-If you're a developer of `taproot-assets` and are interested in attaching your
-signature to the final release archive, the manifest MUST be signed in a manner
-that allows your signature to be verified by our verify script
-`scripts/verify-install.sh`. 
+# Attesting a Manifest File
 
-Assuming you've done a local build for _all_ release targets, then you should
-have a file called `manifest-TAG.txt` where `TAG` is the actual release tag
-description being signed. The release script expects a particular file name for
-each included signature, so we'll need to modify the name of our output
-signature during signing.
+If you're a developer of `taproot-assets` and want to attest a
+build manifest, the manifest MUST be signed in a manner that
+allows your signature to be verified by our verify script
+`scripts/verify-install.sh`.
+
+You will first need to make a pull request adding your signing public
+key, named `$SIGNER.asc`, to the scripts/keys subdirectory. Then, build
+the release artifacts for *all* targets as described in [Building
+a Release](#building-a-release). This will include the checksummed
+artifacts manifest, `manifest-$TAG.txt`.
+
+To generate a detached signature for the manifest, perform the following:
 
-Assuming `USERNAME` is your current nick as a developer, then the following
-command will generate a proper signature:
 ```shell
-$  gpg --detach-sig --output manifest-USERNAME-TAG.sig manifest-TAG.txt
+$ gpg --detach-sig --output "manifest-$SIGNER-$TAG.sig" "manifest-$TAG.txt"
 ```
+
+and then upload it to the 'Assets' of the target release [on
+GitHub][ghrelease].
+
+[ghrelease]: https://github.com/lightninglabs/taproot-assets/releases

make/builder.Dockerfile

@@ -1,10 +1,10 @@
-FROM golang:1.24.9-bookworm
+FROM golang:1.24.9-bookworm@sha256:737b40b61ce956d738bed59f18ba854d8d67e7a4c4fa63f64437f4c70247ac5b
 
-MAINTAINER Olaoluwa Osuntokun <laolu@lightning.engineering>
+LABEL maintainer="Olaoluwa Osuntokun <laolu@lightning.engineering>"
 
 # Golang build related environment variables that are static and used for all
 # architectures/OSes.
-ENV GODEBUG netdns=cgo
+ENV GODEBUG=netdns=cgo
 ENV CGO_ENABLED=0
 
 # Set up cache directories. Those will be mounted from the host system to speed

make/release_flags.mk

@@ -9,7 +9,6 @@ VERSION_CHECK = ./scripts/release.sh check-tag "$(VERSION_TAG)" "$(VERSION_GO_FI
 endif
 
 DOCKER_RELEASE_HELPER = docker run \
-  -it \
   --rm \
   --user $(shell id -u):$(shell id -g) \
   -v $(shell pwd):/tmp/build/taproot-assets \

scripts/get-git-tag-name.sh

@@ -15,6 +15,12 @@ get_git_tag_name() {
       exit 1
   fi
 
+  grep_version=$(grep --version)
+  if [[ ! "$grep_version" =~ "GNU grep" ]]; then
+    echo "GNU grep is required, but cannot be found."
+    exit 1
+  fi
+
   # Read and parse the version fields. We interpret these fields using regex
   # matching which effectively serves as a basic sanity check.
   local app_major

scripts/release.sh

@@ -28,7 +28,7 @@ function reproducible_tar_gzip() {
   # work properly.
   tar_version=$(tar --version)
   if [[ ! "$tar_version" =~ "GNU tar" ]]; then
-    if ! command -v "gtar"; then
+    if ! command -v "gtar" &>/dev/null; then
       echo "GNU tar is required but cannot be found!"
       echo "On MacOS please run 'brew install gnu-tar' to install gtar."
       exit 1
@@ -56,6 +56,13 @@ function reproducible_zip() {
   # Pin down file name encoding and timestamp time zone.
   export TZ=UTC
 
+  # 'zip' is commonly missing on a number of Linux distributions, and
+  # may be missing in containers.
+  if ! command -v "zip" &>/dev/null; then
+    echo "zip(1) is required, but cannot be found."
+    exit 1
+  fi
+
   # Set the date of each file in the directory that's about to be packaged to
   # the same timestamp and make sure the same permissions are used everywhere.
   chmod -R 0755 "${dir}"
@@ -125,6 +132,19 @@ function build_release() {
   local buildtags=$3
   local ldflags=$4
   local goversion=$5
+  local shasum_cmd="shasum -a 256"
+
+  # If we don't have a Perl installation available, then check for
+  # coreutils's 'sha256sum'.
+  if ! command -v "shasum" &>/dev/null; then
+    if ! command -v "sha256sum" &>/dev/null; then
+      echo "Either a Perl or GNU coreutils installation is required."
+      exit 1
+    else
+      # coreutils is installed, so use sha256sum instead.
+      shasum_cmd="sha256sum"
+    fi
+  fi
 
   # Check if the active Go version matches the specified Go version.
   active_go_version=$(go version | awk '{print $3}' | sed 's/go//')
@@ -159,7 +179,7 @@ required Go version ($goversion)."
   tar -xf "${package_source}.tar" -C ${package_source}
   rm "${package_source}.tar"
   reproducible_tar_gzip ${package_source}
-  mv "${package_source}.tar.gz" "${package_source}-$tag.tar.gz" 
+  mv "${package_source}.tar.gz" "${package_source}-$tag.tar.gz"
 
   for i in $sys; do
     os=$(echo "$i" | cut -f1 -d-)
@@ -189,7 +209,7 @@ required Go version ($goversion)."
 
     # Add the hashes for the individual binaries as well for easy verification
     # of a single installed binary.
-    shasum -a 256 "${dir}/"* >> "manifest-$tag.txt" 
+    $shasum_cmd "${dir}/"* >> "manifest-$tag.txt"
 
     if [[ $os == "windows" ]]; then
       reproducible_zip "${dir}"
@@ -199,7 +219,7 @@ required Go version ($goversion)."
   done
 
   # Add the hash of the packages too, then sort by the second column (name).
-  shasum -a 256 $PACKAGE-* vendor* >> "manifest-$tag.txt"
+  $shasum_cmd $PACKAGE-* vendor* >> "manifest-$tag.txt"
   LC_ALL=C sort -k2 -o "manifest-$tag.txt" "manifest-$tag.txt"
   cat "manifest-$tag.txt"
 }