Run Test Continuously

Previous: Building Qt in a local Windows VM

With the first build of Qt done and a first test executed successfully on the VM using the build job, let's have a closer look at what options the run command provides, and what other useful minicoin commands there are.

Flaky Test Testing

There are tests in Qt that fail sometimes, because they depend on unstable conditions of the runtime environment. Some tests are timing sensitive, some rely on the position of the mouse cursor, or the network, or other components that are unpredictable. Confirming that they are flaky might require some stress testing of the respective test function. One such flaky test is the aptly named QWindow::spuriousMouseMove.

$ minicoin run build --target tst_qwindow_check --testargs spuriousMouseMove

This is equivalent to doing the following locally

$ cd ~/qt/dev-build # local build tree for dev
$ TESTARGS=spuriousMouseMove ninja tst_qwindow_check

But running a flaky test only once might not give us a large enough sample size, so let's run it 10 times instead.

$ minicoin run --repeat 10 build --target tst_qwindow_check --testargs spuriousMouseMove windows11

Here we use the repeat functionality of the run command. The repeat loop is executed on the VM guest, so we don't have to pay the overhead of creating an ssh connection to the guest 10 times.

At the end of this loop, the run command will inform us for how many of the 10 runs the build job returns success (i.e. exit code zero).

Testing as you type

As a recap, we have declared a mutagen role definition for all machines to make sure that the file system is continuously synced from the host machine to the VMs. So any change we make locally is almost instantly also synced to the virtual machine. It would be nice to kick off a build test on the VM as soon as we make changes.

$ minicoin run --fswait build --target Gui windows11

The fswait option tells minicoin to wait for file system changes after each completion of the build job, and to run the job again, with a few seconds delay once a file system change is detected. The wait-and-repeat logic happens entirely on the guest VM, so there is no overhead for recreating SSH connections to the VM after each iteration.

Especially when implementing platform specific code or fixing issues that only occur on e.g. Windows or with a specific compiler while working on a different host system, this can give very quick turnaround times (but also very high CPU consumption if you are working on qglobal.h).

Next: Building and testing Qt using AWS