Integration of Parallel running with pipelines....

[v-mwalk]

My current gig is using Azure Devops with a bunch of agent machines.

They are using a stable and well structured Cucumber-JS based framework for their functional regression test suites (API-based and F/E-based applications). However, with the number of tests running to the many hundreds/thousands for some test suites I have been looking to optimise and use parallel execution.

To enable this, I'm using Azure's parallel running system and Cucumber tags (IE. @thread1, @thread2 etc etc.... Cucumber called by each parallel thread along with the Thread number and Tags used to execute only the tests for that thread). The issue I have found is tests forget to use a tag. Or mistype a tag. And there is no way to easily know if a test is not being executed in pipeline due to a missed/mistyped thread tag.

I'm thinking how the current Cucumber-JS parallel execution mechanism could be integrated into a host devops environment - such as (but not limited to) Azure. When the devops pipeline is executed with a 'parallel' strategy, Azure clones the host execution image to each parallel hoster and off they go executing (Azure controllers know what 'thread' they are on by use of Azure provided JobPositionInPhase/TotalJobsInPhase - hence I use that when building the thread filter Tag on each).

Use of plugins sounds interesting. But would there be enough power in their use to allow thsi? And how would it work?

• The Coordinator would need to be able to communicate with workers using TCP/IP (or whatever way system like Azure provide to allow communication).
• There would need to be a way launch Cucumber-JS telling it if it is a Coordinator (in which case I guess I just looks at the tests in feature files so it can queue them etc...) in a 'distributed' system. It would need a list of 'agents' along with the connection details etc.
• There would need to be a way to launch Cucumber-JS as a 'worker' whereby it just listens for and reacts to coordinator messages (as it does currently; only via TCP/IP or whatever). It would also have to do whatever startup it currently does (Transpile etc...?).

While anything is possible 'just bytes on a disk...', I'm not sure it would be 'easy' to do taking into a number of technical and non-technical issues. But it would very much be quite a game changer..... Any thoughts?

[vitalets]

The issue I have found is tests forget to use a tag. Or mistype a tag. And there is no way to easily know if a test is not being executed in pipeline due to a missed/mistyped thread tag.

Just as an idea: what if run one more worker with --tags "not @thread1 and not @thread2 ..." to filter tests that are not covered by any tag?

[luke-hill]

We wouldn't do this because the rules stipulate that tags do not need to exclude scenarios, it might be you've just removed a tag so don't want to change your profile or you've not currently got any tags that match your rules.

[v-mwalk]

Yeah, we did go down this route to start with. Issue was that some tests are deliberatly tagged to NOT run in pipeline (and indeed some ONLY run in pipeline lol!).

Way I've done it so far is to stipulate that all feature files have a tag @testsuite<n>. A pre-execution script flips through the feature files and checks they all comply (it also does a count check to get a rough idea of loading - rough, as it doesnt check number of tests etc...); bombing out if any dont. The other tags (@ignore, @Full, @smoke, @PipelineOnly etc) govern if tests actually execute.

Really, it is just a bit messy (and fails my 'KISS' criteria lol) and Cucumber's handling of parallel testing (using a coordinator to dish out tests dynamically) works really well - only, on a single machine.

Starting to look at how node.js does this - all to do with 'Horizontal Scaling'. Although it looks like it would need to be 'hand rolled' etc...