Create nightly builds

[lidel]

cc / supersede #77 & #78

Motivation

We want to have a setup that constantly proves our release pipeline works as expected. Producing a nightly builds for all dist packages is the best way for identifying issues early and minimizing risk of delays of Stable and RC releases.

This is the main reason for doing this.
Providing nightly builds to the community is a nice side effect, but not something we will explicitly support or advertise anywhere.

Architecture

• We want a daily confirmation that everything can be built, so we want to cut a new release without worrying about CI.
• Nightly builds will be automated via Github Actions, and will reuse as much of the regular release pipeline, as possible.
• Would be nice to have a separate website at nightly.dist.ipfs.io will provide N days of nightly builds, but keeping it limited to github actions is enough for MVP
• Pin artifacts to website cluster for persistence

TODO

Rough plan

• extend ./dist.sh add-version <dist> <version> with suport for a special " nightly" version
  • nightly should build the latest revision from the default branch (don't assume it is master)
  • artifact names should have nightly-2021-07-16-9599ad5 instead of regular versions like v0.9.0 (build date and first 7 characters of git revision used as the source)
  • keep only N nightlies around (eg. 7)
  • use /ipns/nightly.dist.ipfs.io or /ipns/dist.ipfs.io as the base root (update: we want nightly test that adds patch on top of existing website, so original /ipns/dist.ipfs.io has to be used)
  • idempotent within each day: skip build if the revision was already built for today
• set up DNS infra for nightly.dist.ipfs.io, add DNSIMPLE_TOKEN to ipfs/distributions repo to enable update via CI (prior art)
• add a new cron workflow to Actions that builds a nightly every day.
  • Create .github/workflows/nightly.yml
    • Run make publish pipeline just like main.yml, but find a way to build nightly for every package defined in ./dists directory
      • echo "nightly_2021-07-16" >> dists/go-ipfs/versions and then make publish producing nightly binaries in ./releases
        • this has cool property of enabling us to make all_dists (build all binaries) for each nightly in separate, parallel job, and then merge ./dists and ./releases before running make publish in final step (generating website and putting everything on IPFS as a patch on top of DIST_ROOT)
      • if any package failed in the middle of the pipeline, the whole nightly job should fail (but we should set the "failed" flag at the very end, so we see all packages that are broken, not only the first one)
      • leverage DIST_ROOT=/ipfs/Qm.. for starting with an empty or customised root for nightlies without waiting for dnslink at nightly.dist.ipfs.io

[BigLep]

@lidel : does this supercede ipfs/kubo#8250 ? One item listed there that I want to ensure we have is getting notification if a nightly build fails (or at least if we get 2 consecutively).

[lidel]

@BigLep yes, this issue is essentially implementation of the process you described.
Added notification requirement to the MVP definition. TBD: could be email to mailing list, could be slack message – lmk which is preferable.

[iand]

@lidel do we want to build a nightly even if nothing has been committed since the previous nightly build? I think for low churn repos we'll end up with N=7 identical builds with unique names.

[lidel]

@iand yes, we want to always build, even if nothing changed to keep confirming that the build pipeline works.
even if code did not change, external factors such as apple notarization may break (eg. start rejecting specific package) and we want to catch that early.