Add a possibility to run tests simultaneously

[filipleple]

The problem you're addressing (if any)

We run hours long tests continuously probing the cpu frequency, then we reboot and run hours long tests continuously probing the cpu temperature, then we reboot and run hours long tests continuously checking if the platform hasn't rebooted, then the same all over but under load etc....

Describe the solution you'd like

Boot the platform once, probe frequency, temperature and uptime and then save the results to appropriate test cases (CPF00n, CPT00n, STB00n). Then repeat under load.

EXTRA: It could also be possible to SSH into the platform in the meantime and do some stateless tests, i.e. not requiring options to be toggled, working on default settings, e.g. dmidecode. Such tests would be tagged, and there would be a mode of running tests on the side that only uses SSH and doesn't do any power toggling

Where is the value to a user, and who might that user be?

At least 3x speedup of the longest tests.

Furthermore, that way it's way more convenient to set up overnight testing. You would only need to see the first iteration of such combined test to know whether frequency, temperature and stability are checked correctly - no chance of the next test over crashing, ruining your overnight regression.

Describe alternatives you've considered

Manually probing multiple things at once. But we have it all automated, the tests just need to be coordinated better.

Additional context

No response

[filipleple]

@BeataZdunczyk What exactly would need to be done is:

• Take the following test suites:
  • CPU frequency
  • CPU temperature
  • STB00.. Verify if no reboot occurs in the OS
• Each of them just executes a probe command every X minutes until it's been 1 hour or so
• Copy those commands and paste them into one test that runs all of them in sequence and then waits the X minutes
• Ideally log the probed values to the console on the go
• Don't fail and stop the entire process when either of the commands returns a bad value, just make note of it
• After the 1 hour has passed, collect the results and pass or fail corresponding test cases
  • This might get tricky, not sure how easy is it to pass and modify global variables between test cases in Robot

Of course, this would still need to be done separately for with/without load, Windows/Ubuntu. For each 4 cases (with/without load, Windows/Ubuntu) we take three 1-hour-long tests and turn them into one 1-hour-long test, saving 2 hours. That would mean we save 4*2=8 hours by doing this, in each regression testing session.

[philipanda]

@BeataZdunczyk What exactly would need to be done is:

* Take the following test suites:
  
  * CPU frequency
  * CPU temperature
  * STB00.. Verify if no reboot occurs in the OS

* Each of them just executes a probe command every X minutes until it's been 1 hour or so

* Copy those commands and paste them into one test that runs all of them in sequence and then waits the X minutes

* Ideally log the probed values to the console on the go

* Don't fail and stop the entire process when either of the commands returns a bad value, just make note of it

* After the 1 hour has passed, collect the results and pass or fail corresponding test cases
  
  * This might get tricky, not sure how easy is it to pass and modify global variables between test cases in Robot

Of course, this would still need to be done separately for with/without load, Windows/Ubuntu. For each 4 cases (with/without load, Windows/Ubuntu) we take three 1-hour-long tests and turn them into one 1-hour-long test, saving 2 hours. That would mean we save 4*2=8 hours by doing this, in each regression testing session.

Multiply that by 3 for Novacustom laptops where we test these test suites on different power sources: AC, USB-C PD, and battery

[philipanda]

I've been thinking about different ways of implementing this:

• a python library with global state that runs parallel workers that measure the stats over time
• running multiple test cases using pabot, creating some kind of sync mechanism between the processes, a file or a socket
• running all measurements in init.robot of a new test module and only adding checks for the gathered data in the tests

I don't think any alternative would be considerably better than the idea to just add a pseudo-test case before every block of checks. The pseudo test case would be what actually gathers all the data. They would log the results to the console too.

These pseudo test cases would need to either have some special name or ID that is not interpreted by the CI scripts that guard osfv-dashboard integrity, or we could add them to osfv-dashboard and remember that they are not real test cases. I think the former is a much better option.

The order of test case execution would look like:

_PSEUDO_Background Stability Data Init (AC) (no load) (Ubuntu)              | PASS|
CPF003.201 CPU not stuck on initial frequency (Ubuntu) (AC)                 | PASS|

_PSEUDO_Background Stability Data Gathering (no load) (Ubuntu)              | PASS|
CPF007.201 CPU runs on expected frequency (Ubuntu)                          | PASS|
CPT001.201 CPU temperature without load (Ubuntu)                            | PASS|
STB001.201 Verify if no reboot occurs in the OS (Ubuntu)                    | PASS|

_PSEUDO_Background Stability Data Gathering (cpu load) (Ubuntu)             | PASS|
CPF007.201 CPU with load runs on expected frequency (Ubuntu)                | PASS|
CPT001.201 CPU temperature after stress test (Ubuntu)                       | PASS|

For laptops all of this would be duped a couple times to take the power source into account.

Drawback: there would be no clean way to run a single test case. There is no way to know in the test case if robot was called with a test case name filter -t option or with a custom set of tags -i option.
A way to work this around is to again add more pseudo test cases. One pseudo test case for every real test case.
Something like:

_PSEUDO_CPT001.201 CPU temperature after stress test (Ubuntu) 
    - TESTS_TO_RUN += "CPT001.201"
_PSEUDO_Background Stability Data Gathering (cpu load) (Ubuntu) 
    - TESTS_TO_RUN contains "CPT001.201" - must gather temperature data 
CPT001.201 CPU temperature after stress test (Ubuntu) 

I am confident something like that would work great. It looks cursed though.

I would say that tests that could be parallelised like that should be moved into a separate module.
dasharo-parallelable or something like that.
One master test suite could manage multiple test suites and allow them to be run much much faster

[philipanda]

Created a PR that shows a proof of concept for determining whether a test should be run in #1130