Canary deployment does not always reflect latest merged PR

[victorlin]

I just merged two PRs in this order: #914, #922. next.nextstrain.org reflected the latest #922 briefly before the slower build of #914 completed and overrode the latest changes from #922.

Deploy is done as the last step of CI (which is also CD in this case). You can see that the workflows started around the same time but #914's was slower than #922's:

Possible solutions

1. Use jobs.<job_id>.concurrency
   • I think this can be difficult because the actual deploy step is quick and will not overlap in most scenarios. The overlapping happens during build but concurrency shouldn't be checked on that job – it'd still be good to have a complete build for each merged PR. Maybe having a separate build but then we wouldn't be deploying the same exact build that's being tested.

[tsibley]

We do use concurrency controls already:

nextstrain.org/.github/workflows/ci.yml

Lines 126 to 127 in 6b7f354

	# Only one "deploy" job per Heroku app at a time.
	concurrency: deploy:${{ inputs.heroku-app || 'nextstrain-canary' }}

but yeah, it's only on the job where it's not enough to prevent this. Note that Heroku's own builds suffer from the same failure mode (I've seen it more than once).

it'd still be good to have a complete build for each merged PR.

Why? I agree it'd be good to have a complete build for each PR while it's a PR, but not each merged PR. Once something's merged, we've serialized it onto master and it would be fine IMO for master builds to be serial too and use concurrency with cancel-in-progress: true.

[victorlin]

Good point. Maybe it'd be simpler to set concurrency on the workflow level, specifying at most one active run per branch?

[tsibley]

Sure. That works, though for (unmerged) PRs we might not want to wait around for previous CI runs to complete before launching new ones on new pushes?

[victorlin]

How about using cancel-in-progress: true so that previous CI runs will get canceled on new pushes?

[tsibley]

Sure, we can try it out and see how we like it.