Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add govulncheck to CI #6963

Merged
merged 29 commits into from
Jul 11, 2023
Merged
Show file tree
Hide file tree
Changes from 25 commits
Commits
Show all changes
29 commits
Select commit Hold shift + click to select a range
86f46ed
Add govulncheck and update patch version of golangci-lint
pgporada Jun 28, 2023
5f36536
Split arguments onto newlines for readability
pgporada Jun 28, 2023
6ae89b2
Add govulncheck to lint runs
pgporada Jun 28, 2023
50d07a1
Add govulncheck setup script to Dockerfile and use less layers
pgporada Jun 28, 2023
fa25ae3
Update golangci-lint to latest version and add a todo to remove the g…
pgporada Jun 28, 2023
c90d70c
Setup govulncheck local filesystem API structure
pgporada Jun 28, 2023
a2ece93
Run govulncheck only on github workers
pgporada Jun 30, 2023
0c56436
Update boulder-tools readme for go version updates
pgporada Jun 30, 2023
d46118a
Move govulncheck into separate github action
pgporada Jun 30, 2023
ba7a1fe
Use proper matrix
pgporada Jun 30, 2023
0557cc5
Update job name
pgporada Jun 30, 2023
6d76aef
Move govulncheck as a new job in the boulder-ci workflow
pgporada Jun 30, 2023
631ea05
give github a nudge
aarongable Jul 3, 2023
2b2261a
Remove govulncheck-action because the internal setup-go can't yet dow…
pgporada Jul 5, 2023
a41e6f3
Fix erroring out due to cannot query module because mod=vendor
pgporada Jul 5, 2023
343f1d0
Container shenanigans
pgporada Jul 5, 2023
271c8bd
Debugging
pgporada Jul 5, 2023
4ef7bb3
Finished debugging, probably
pgporada Jul 5, 2023
64c8d9a
Reload the github env
pgporada Jul 5, 2023
8385235
Remove extra newline
pgporada Jul 5, 2023
40dfcf9
It's a day, that's for sure
pgporada Jul 5, 2023
379b435
Override GOFLAGS inside the container during govulncheck install only
pgporada Jul 5, 2023
0ca8a92
Override GOBIN for the install too
pgporada Jul 5, 2023
6ca0506
Matrix wasn't being used during govulncheck steps, now it is
pgporada Jul 5, 2023
fc952ac
Address comments
pgporada Jul 5, 2023
f7d2ddd
Add govulncheck to utility status check job
pgporada Jul 5, 2023
5ebd869
Use go run instead of spawning a bash shell and then invoking go
pgporada Jul 10, 2023
4fdd71a
Modify this file in a future PR
pgporada Jul 10, 2023
d09895e
Add comment about unsetting GOFLAGS
pgporada Jul 10, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
53 changes: 53 additions & 0 deletions .github/workflows/boulder-ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -105,6 +105,59 @@ jobs:
- name: "Run Test: ${{ matrix.tests }}"
run: ${{ matrix.tests }}

govulncheck:
runs-on: ubuntu-20.04
strategy:
# When set to true, GitHub cancels all in-progress jobs if any matrix job fails. Default: true
fail-fast: false
matrix:
# Add additional docker image tags here and all tests will be run with the additional image.
BOULDER_TOOLS_TAG:
- go1.20.5_2023-06-20
- go1.21rc2_2023-06-21

env:
# This sets the docker image tag for the boulder-tools repository to
# use in tests. It will be set appropriately for each tag in the list
# defined in the matrix.
BOULDER_TOOLS_TAG: ${{ matrix.BOULDER_TOOLS_TAG }}

steps:
# Checks out your repository under $GITHUB_WORKSPACE, so your job can access it
- uses: actions/checkout@v3
with:
persist-credentials: false

- name: Docker Login
# You may pin to the exact commit or the version.
# uses: docker/login-action@f3364599c6aa293cdc2b8391b1b56d0c30e45c8a
uses: docker/login-action@v2.1.0
with:
# Username used to log against the Docker registry
username: ${{ secrets.DOCKER_USERNAME}}
# Password or personal access token used to log against the Docker registry
password: ${{ secrets.DOCKER_PASSWORD}}
# Log out from the Docker registry at the end of a job
logout: true
continue-on-error: true

pgporada marked this conversation as resolved.
Show resolved Hide resolved
# Print the env variable being used to pull the docker image. For
# informational use.
- name: Print BOULDER_TOOLS_TAG
run: echo "Using BOULDER_TOOLS_TAG ${BOULDER_TOOLS_TAG}"

# Pre-pull the docker containers before running the tests.
- name: docker compose pull netaccess
run: docker compose pull netaccess

# Enable https://github.com/golang/go/wiki/LoopvarExperiment if we're on
# go1.21rc2 or higher. This experiment value is unknown in lower versions.
- if: startsWith(matrix.BOULDER_TOOLS_TAG, 'go1.21')
run: echo "GOEXPERIMENT=loopvar" >> "$GITHUB_ENV"

- name: Install and run govulncheck
run: docker compose run --use-aliases netaccess bash -c 'GOFLAGS="" GOBIN=/usr/local/bin go install golang.org/x/vuln/cmd/govulncheck@latest && govulncheck ./...'
aarongable marked this conversation as resolved.
Show resolved Hide resolved
pgporada marked this conversation as resolved.
Show resolved Hide resolved

# This is a utility build job to detect if the status of any of the
# above jobs have failed and fail if so. It is needed so there can be
# one static job name that can be used to determine success of the job
Expand Down
7 changes: 3 additions & 4 deletions test/boulder-tools/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -11,13 +11,12 @@ COPY requirements.txt /tmp/requirements.txt
COPY boulder.rsyslog.conf /etc/rsyslog.d/
COPY build.sh /tmp/build.sh
RUN /tmp/build.sh

# Note: This arg and env variable should only be set _after_ build.sh installs
# necessary apt packages. Otherwise, we will reinstall those apt packages for
# each Go version, rather than reusing a cached layer.
ARG GO_VERSION
ENV GO_VERSION=$GO_VERSION
COPY install-go.sh /tmp/install-go.sh
RUN /tmp/install-go.sh
RUN sed -i '/imklog/s/^/#/' /etc/rsyslog.conf
RUN sed -i '/$ActionFileDefaultTemplate/s/^/#/' /etc/rsyslog.conf
RUN sed -i '/$RepeatedMsgReduction on/s/^/#/' /etc/rsyslog.conf
RUN /tmp/install-go.sh && \
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why combine the rsyslog editing steps with the install-go step? Though I do like combining the different sed invocations into a single invocation.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

RUN, COPY, and ADD create new layers which use some amount of disk space. Having less layers is more efficient disk space-wise. There's further optimizations that can be done with a multi-stage build, but that's outside the scope of this PR. Hell, even this change is now outside the scope of this PR because I changed strategies mid-way through.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Layers prior to this change
Screenshot from 2023-07-10 16-40-26

Layers after this change
Screenshot from 2023-07-10 16-49-52

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point, and that's why for instance we put a bunch of stuff inside build.sh rather than in RUN commands directly. Perhaps the sed commands should go inside build.sh? That makes more sense to me logically than clustering them with install-go.sh. It would probably be interesting to look at the git history and see if there's a specific reason they're in the Dockerfile rather than build.sh.

sed -i -e '/imklog/s/^/#/' -e '/$ActionFileDefaultTemplate/s/^/#/' -e '/$RepeatedMsgReduction on/s/^/#/' /etc/rsyslog.conf
3 changes: 2 additions & 1 deletion test/boulder-tools/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,8 @@ to our workflow:
2. We run the `tag_and_upload.sh` script to build, tag, and upload
a `boulder-tools` image for each of the `GO_VERSIONS`.
3. We update `.github/workflows/boulder-ci.yml` to add the new image tag(s).
4. We update `docker-compose.yml` to update the default image tag (optional).
4. We update the remaining `.github/workflows/` yaml files that use a `GO_VERSION` matrix with the new version of Go.
5. We update `docker-compose.yml` to update the default image tag (optional).

After some time when we have spot checked the new Go release and coordinated
a staging/prod environment upgrade with the operations team we can remove the
Expand Down
3 changes: 1 addition & 2 deletions test/boulder-tools/install-go.sh
Original file line number Diff line number Diff line change
Expand Up @@ -18,8 +18,7 @@ go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@v1.2.0
go install github.com/rubenv/sql-migrate/...@v1.1.2
go install golang.org/x/tools/cmd/stringer@latest
go install github.com/letsencrypt/pebble/cmd/pebble-challtestsrv@master
go install github.com/golangci/golangci-lint/cmd/golangci-lint@v1.51.0
go install github.com/golangci/golangci-lint/cmd/golangci-lint@v1.53.3

go clean -cache
go clean -modcache

6 changes: 4 additions & 2 deletions test/boulder-tools/tag_and_upload.sh
Original file line number Diff line number Diff line change
Expand Up @@ -29,9 +29,11 @@ build_and_push_image() {
echo "Building boulder-tools image ${TAG_NAME}"

# build, tag, and push the image.
docker buildx build --build-arg "GO_VERSION=${GO_VERSION}" \
docker buildx build \
--build-arg "GO_VERSION=${GO_VERSION}" \
--progress plain \
--push --tag "${TAG_NAME}" \
--push \
--tag "${TAG_NAME}" \
--platform "${PLATFORMS}" \
.
}
Expand Down