Support Asynchronous pipeline test runs to eliminate Pipeline agent waiting

[aglasson]

Is your feature request related to a problem? Please describe.
Our organisation is currently running Microsoft Playwright Testing (hosted service) in Azure DevOps Pipelines and the solution requires the agent to invoke Playwright testing and wait for it to complete and return the results. This means tying up an agent concurrency slot and for self hosted agents are paying for runtime sometimes for hours when they are essentially doing nothing waiting for the Playwright service to finish.

In our organisation we like to have pipeline Agents (we use Containers for Linux and Managed DevOps Pools for Windows) capped at 1 hour runtime to reduce the risk of massive cost blowouts if we get hung pipelines etc. The playwright jobs become challenging because some of them can run for much longer. This wouldn't be a problem if Hosted Playwright didn't depend on self-hosted agent instance timeouts.

Describe the solution you'd like

1. It is completely acceptable to require an agent to compile the test cases and submit the playwright job to the service. It would be good if the node command line Playwright actions (our pipeline task) could complete successfully simply upon submission of the job to the service. Hypothetically a command line parameter --submit-job-only.

2. Playwright could have a job status API that supports webhook / call back functionality.
   The Azure DevOps agentless pipeline task InvokeRESTAPI@1 for example has the waitForCompletion property which allows the task to wait for a callback to the pipelines REST API.
   Read the section:

   To signal completion, the external service should POST completion data to the following pipelines REST endpoint.

3. Upon API callback of job completion the pipeline would continue with agent-based activities to retrieve the output of the job and handle local reporting of tests successes, failures, or otherwise. Hypothetically a playwright command line parameter --retrieve-job-only or perhaps a separate Azure Task could be defined for Playwright Hosted service job retrieval?

This is just one hypothetical implementation. I'm sure there are other ways, perhaps even simpler, to implement such a solution.

Describe alternatives you've considered
Have considered what command line options or APIs exist for invoking Playwright hosted jobs without waiting for completion and nothing suitable seems to exist.

Additional context
We have engaged Microsoft Support on this topic and they concluded that Asynchronous hosted Playwright Testing jobs are not possible. They suggest we raise a request for this functionality.

I have searched existing enhancement requests (issues) in this GitHub repo and not found anything similar.