-
Notifications
You must be signed in to change notification settings - Fork 32
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: create nightly builds #369
Merged
Merged
Changes from all commits
Commits
Show all changes
36 commits
Select commit
Hold shift + click to select a range
cd78bdf
wip: signing poc with hardcoded dist details
lidel 230d1cd
feat: create nightly builds
iand 1bb2f20
Ensure nightly dates use utc
iand 7208eae
Allow manual invocation of nightly workflow
iand 42531b6
Keep last 7 nightly versions in published versions file
iand 18ca56e
Touch existing versions file to create it
iand 4870454
Create output dir earlier
iand 8e11abe
wip: debugging pinning to cluster
lidel 2cb0822
Publish after nightly build
iand f3c456d
wip: wait for 2 cluster confirmations
lidel 3e9a8e2
wip: speed up build
lidel 796aaae
wip: PR preview link
lidel ed656e5
wip: no-op build
lidel 7d222b5
refactor: use GO_IPFS_VER in dockerized build
lidel f115f93
docs: DNSLink discussion
lidel 9acff7e
refactor: simplify
lidel 5ea24dc
refactor: avoid polluting ./releases
lidel 5df807b
refactor: respect nightly cut-off date
lidel 716ebaf
wip: see how long all nightlies take to build and pin
lidel 6e513af
fix: dockerized build on ci
lidel baa201f
fix: vtag dists like fs-repo-migrations/fs-repo-*
lidel a2e8f26
wip: run each dist in matrix
lidel 87c0b11
refactor: scripts/ci/setup-ipfs.sh
lidel 9544654
refactor: scripts/ci/pin-to-cluster.sh
lidel c239cfb
fix: quic perf
lidel ef7843a
refactor: sign-new-macos-releases.sh
lidel 835df84
refactor: github-preview-link.sh
lidel 1c9b839
test: signed release built for ipfs-update@v1.7.1
lidel 00dc19b
Merge branch 'feat/signed-ci-build' into feat/issue-363-nightlies
lidel e2570b8
refactor: reuse ./scripts/ci
lidel 947f37d
fix: nightly cleanup
lidel 5277083
refactor: deduplicate pinning
lidel 843feda
feat: fill issue for every broken nightly build
lidel 08caa11
fix: github status link
lidel dc8b64e
Merge branch 'master' into feat/issue-363-nightlies
lidel 6a29155
chore: restore target trigger (cron and manual)
lidel File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,106 @@ | ||
name: Nightly | ||
|
||
on: | ||
workflow_dispatch: | ||
inputs: | ||
dist_root: | ||
description: 'DIST_ROOT' | ||
required: true | ||
default: '/ipns/dist.ipfs.io' | ||
schedule: | ||
- cron: '0 5 * * *' # UTC | ||
|
||
env: | ||
DIST_ROOT: ${{ github.event.inputs.custom_dist_root || '/ipns/dist.ipfs.io' }} # content root used for calculating diff to build | ||
GO_IPFS_VER: 'v0.9.1' # go-ipfs daemon used for chunking and applying diff | ||
CLUSTER_CTL_VER: 'v0.14.0' # ipfs-cluster-ctl used for pinning | ||
|
||
concurrency: | ||
group: nightly | ||
cancel-in-progress: true | ||
|
||
jobs: | ||
prepare-matrix: | ||
runs-on: ubuntu-latest | ||
steps: | ||
- uses: actions/checkout@v2 | ||
- id: set-matrix | ||
run: echo "::set-output name=matrix::$(jq -nc '$ARGS.positional' --args $(ls ./dists -1))" | ||
outputs: | ||
matrix: ${{ steps.set-matrix.outputs.matrix }} | ||
|
||
build: | ||
runs-on: "ubuntu-latest" | ||
needs: prepare-matrix | ||
strategy: | ||
fail-fast: false | ||
matrix: | ||
dist_name: ${{ fromJson(needs.prepare-matrix.outputs.matrix) }} | ||
steps: | ||
- uses: actions/checkout@v2 | ||
- name: Setup IPFS | ||
run: ./scripts/ci/setup-ipfs.sh | ||
env: | ||
CLUSTER_USER: ${{ secrets.CLUSTER_USER }} | ||
CLUSTER_PASSWORD: ${{ secrets.CLUSTER_PASSWORD }} | ||
timeout-minutes: 5 | ||
- run: cd ./dists/${{ matrix.dist_name }} && make nightly | ||
- run: ./dockerized make publish | ||
- name: Create issue if build failed | ||
uses: actions/github-script@v4 | ||
if: ${{ failure() }} | ||
with: | ||
script: | | ||
const title = 'Nightly build failed for ${{ matrix.dist_name }}' | ||
const body = '${{ matrix.dist_name }} failed to build from the latest commit: https://github.com/ipfs/distributions/actions/runs/${{ github.run_id }}' | ||
const opts = { owner: context.repo.owner, repo: context.repo.repo } | ||
const response = await github.search.issuesAndPullRequests({ | ||
q: `repo:ipfs/distributions is:issue is:open in:title ${title}` | ||
}) | ||
console.log('github.issuesAndPullRequests', response) | ||
let link | ||
if (response.data.items.length === 0) { | ||
const created = await github.issues.create({ ...opts, title, body, | ||
labels: ['kind/bug', 'need/triage'] | ||
}) | ||
console.log('no open issues, created a new one', created) | ||
link = created.data.html_url | ||
} | ||
for (const issue of response.data.items) { | ||
if (issue.title !== title) continue | ||
console.log('found existing open issue', issue) | ||
const created = await github.issues.createComment({ ...opts, | ||
issue_number: issue.number, | ||
body | ||
}) | ||
console.log('commented on existing open issue', created) | ||
link = created.data.html_url | ||
} | ||
await github.repos.createCommitStatus({ ...opts, | ||
sha: '${{ github.sha }}', | ||
state: 'error', | ||
target_url: link, | ||
context: 'Problem with ${{ matrix.dist_name }}', | ||
description: 'See details in the linked issue' | ||
}) | ||
- name: Inspect git status and contents of ./releases | ||
run: git status && ls -Rhl ./releases | ||
- name: Read CID of updated DAG | ||
id: cid-reader | ||
run: echo "::set-output name=CID::$(tail -1 ./versions)" | ||
- name: Pin new website to ipfs-websites.collab.ipfscluster.io | ||
run: ./scripts/ci/pin-to-cluster.sh | ||
env: | ||
PIN_CID: ${{ steps.cid-reader.outputs.CID }} | ||
PIN_NAME: "ipfs/distributions/nightly/${{ matrix.dist_name }}" | ||
PIN_ADD_EXTRA_ARGS: "--expire-in 168h" | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. ℹ️ this pins nightly artifacts only for a week |
||
CLUSTER_USER: ${{ secrets.CLUSTER_USER }} | ||
CLUSTER_PASSWORD: ${{ secrets.CLUSTER_PASSWORD }} | ||
timeout-minutes: 60 | ||
- name: Update PR status with preview link | ||
run: ./scripts/ci/github-preview-link.sh | ||
env: | ||
GITHUB_TITLE: "Preview for ${{ matrix.dist_name }}" | ||
CONTENT_PATH: "/ipfs/${{ steps.cid-reader.outputs.CID }}/${{ matrix.dist_name }}" | ||
GIT_REVISION: ${{ github.sha }} | ||
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ℹ️ Third-party turn-key solutions exists, but I decided to give
actions/github-script
a try. It is an official action provided by GitHub, provides JS porcelain for running arbitrary API calls.Debugging is a bit painful, but being written in JS could make it more dev friendly than bash+curl, allowing more people to maintain/extend.
actions/github-script
is extremely powerful, only caveat is that it needs a personal access token to do anything outside the current repo.We may leverage it even more for developer flow automation tailored to our needs. For example, it could automatically create/label issues and add them to specific column in specific project.