Avoid duplicate CI runs in pull requests (push+PR tests)

[bhess]

not for this PR, but the push tests and pr tests really need to see each other as we're running every test twice, and given these are large runners, it is burning 2x the money needed :)

Originally posted by @ryjones in #2285 (comment)

[baentsch]

Beyond the money burned, this also violates open-quantum-safe/tsc#5. BTW, worthwhile bringing that issue to completion (by way of documentation everyone can follow)? Further, it seems the "[skip ci]" moniker to maximally save CI work isn't fully honoured either any more (some "CodeQL" analysis runs for minutes without any reason I can see -- anyone able to explain or even fix?).

[baentsch]

Another observation looking at this with some distance (and observing current PRs): The code coverage testing seems to be running longest (> 1h) -- is this sensible to do on every PR? I'd have thought one only runs this regularly (say once a week) to ascertain the project gets better over time and not on every PR. Or did some policy get agreed that code coverage may not drop on any PR? If so, can someone please point me there? @dstebila ?

[baentsch]

And yet another observation watching an apparently trivial PR gobble up many hours of CI runtime: SLH-DSA testing is pretty extensive apparently and currently in its 82nd minute of execution:

--> Is this really sensible & necessary? What about moving these slow tests for this algorithm to the weekly test run?

[bhess]

open-quantum-safe/tsc#5. BTW, worthwhile bringing that issue to completion (by way of documentation everyone can follow)?

Yes, I'll prepare a proposal for the documentation ahead of the next TSC meeting.

[dstebila]

--> Is this really sensible & necessary? What about moving these slow tests for this algorithm to the weekly test run?

Yes, okay with me to move this to the extended or weekly tests. And yes, it would be sensible for us to go over our overall CI configuration to find improvements.

[baentsch]

Yes, okay with me to move this to the extended or weekly tests. And yes, it would be sensible for us to go over our overall CI configuration to find improvements.

Indeed: Overall CI runtime for every new commit to #2290 (which basically just changes some constant strings) is 32 hours, 21 minutes and 11 secs with 3 Windows SLH-DSA tests alone taking more than 2 hours -- each! --> I'd suggest culling these immediately: SLH-DSA also is a "second class" algorithm (has not been subject to CT testing) on a Tier 2 platform, so why test it regularly this "thoroughly"? It doesn't get better just because it's tested more frequently :)

In general, most test time goes to running tests on Windows -- for which running/passing CI is not guaranteed --> Suggest to move all Windows testing (but maybe the most simple test) at least to "merge-to-main" testing or better, weekly testing.

Next: The by far largest runtime goes to SLH-DSA -- again and again. This does not make sense -> Either move this to weekly testing or create a mechanism that activates such tests only if a specific tag is set (along the lines of "release testing").

Proposal target: CI for a commit prior to PR shall run for at most 5mins; CI on a PR at most 15mins and a merge-to-main CI at most 30mins (each elapsed time; running tasks in parallel during this time would be goodness). Anything longer should be moved to weekly testing and out of the "critical path" to enable reasonably quick turn-arounds to PR-review-improve-merge cycle -- and be nice to the environment.

[dstebila]

We now have our initial CI usage guidelines which we can begin to apply to CI usage in liboqs.