Parallelize conformance test runs in CI

[smaye81]

This parallelizes the various test runs for conformance in CI to minimize the duration of the entire process.

Previously, the repo was basically only running the web and conformance tests in parallel. However, each of these have various runs which were running serially. Now, each conformance suite run is split into its own Github workflow. In addition, Node version matrices were added to the Node-related conformance runs to add further coverage without affecting overall time. This results in the following all running concurrently:

Node tests:

• connect-express conformance (Node versions 18, 20, 21)
• connect-fastify conformance (Node versions 18, 20, 21)
• connect-node conformance (Node versions 18, 20, 21)

Web tests:

• connect-web conformance (Chrome)
• connect-web conformance (Firefox)
• connect-web conformance (Safari)

Note that conformance tests are still run as part of the all default goal, so local runs of just make are unaffected. However, the conformance tests are no longer a pre-req of the test target, so simply running test will not run the conformance tests. To do so, run make testconformance.

The above enhancements result in an improvement of approximately 8 minutes on CI runs. The overall run duration is tied to the longest individual workflow, which happens to be connect-web conformance in Chrome at 5 minutes. Therefore, the overall run is around 5 minutes, where previously it was 13 minutes or more.

[timostamm]

Nice improvement!

There are still two cases of sequential runs:

• node runs server and client tests in sequence.
• web runs promise client and callback client in sequence.

Parallelizing both can potentially bring overall runtime below 3min. Considering that we want to move to turborepo, it may not be worth it to further split up the jobs.

But I do wonder why chrome tests takes significantly longer than the two other browsers. Looking at these two runs, chrome takes as long as safari and firefox combined: A, B. Is there something wrong with our setup?

[smaye81]

Our setup should be OK. I checked it against webdriver's docs and it looks fine to me. It's a bit variable too. In this run, the safari tests seem to have taken a bit longer and chrome's tests were around 3m. So, this may be a function of how long it takes the chrome driver to start? Not entirely sure.

Also updated to further parallelize so now we're around 4 minutes or so. I don't love that proliferation of checks now and how the Makefile has kind of blown up, so if you know of a better way around this, lmk. Just feels kind of unwieldy and hard to maintain as-is.

[timostamm]

Thanks for looking into chrome.

Also updated to further parallelize so now we're around 4 minutes or so. I don't love that proliferation of checks now and how the Makefile has kind of blown up, so if you know of a better way around this, lmk. Just feels kind of unwieldy and hard to maintain as-is.

I agree. It's a lot of boilerplate to save 1 minute.

IMO we should punt on it, and parallelize further with #798.

[smaye81]

@timostamm sounds good. I merged in your changes from #1111 and reverted the commits that further parallelized callback vs. promise client and Node clients and servers.

[timostamm]

LGTM!