Building tests independently

[Vekhir]

Meson differentiates between the program to be build and the tests to test the program. The tests are run within the build environment after building the program, but there is the potential to completely separate the tests and allow them to be independently packaged and run to also test the installed program.

The current way of things

Meson currently builds the library during meson build, runs the tests against the local build artifacts via meson test and then installs the library or executable using meson install
So, the workflow looks as follows:

1. Build the library and tests from source to build/
2. Run the tests against the library in build/
   This works fine to check that there are no issues with the code and to be confident before going into packaging and installing.

The proposal

The proposal here is to be able to build the tests independently in a way that, once built, they can be run against the installed package. That is:

1. Build the library from source
2. Install the package to the system.
3. Build the tests from the same source
4. Install the tests to the system
5. Run the tests against the installed package.

Advantages

This does look somewhat more complicated, but has several advantages from a packaging standpoint.

• The first is that the tests actually test what's installed, i.e. that the installation can be found, that dependencies are correct, etc. I don't necessarily expect meson to do all that, instead separately packaged tests allow the packagers to do that.
• The second one is better automation of rebuilding, since the tests can be used to check for compatibility (at least that's their purpose), and if they haven't changed in interface or behaviour, their dependent packages don't need rebuilding.
  More concretely, suppose a package X depends on package P. If we update P, however minor the change, we get P' which isn't identical (being reproducible doesn't affect this). Running the tests for P against P', we can identify that P' has in fact the same behaviour as P (or not) and automatically determine whether we need to rebuild X, e.g. due to ABI changes. The same principle applies when updating a dependency of P or recompiling with a newer/different compiler.
• Lastly, this change allows greater separation of building and testing, in the sense that the resulting binary can be tested after the fact and repeatedly. It also helps to avoid dependency cycles, where the package X is itself a test dependency of P and both need updating. This is rare in C++-land, but quite common with e.g. Haskell, so not impossible.

Drawbacks

Depends on the implementation, and I don't think they are severe, but here are some I thought of:

• Could increase build times, when also checking. This might happen if creating the test package requires the rebuild of the program, but cannot be combined with the original one. In effect, it would around double the build time.
• Increased complexity in dealing with tests, though Meson already has different mechanics and only builds tests when required.
• Potentially breaking changes, when some user interfaces allow for odd behaviour or are otherwise incompatible with this rather different approach. I don't have a concrete case, so maybe there is nothing here, but naturally, this is going to come out during discussion.

Implementation

In terms of implementation, step 1 and 2 are trivial.
AFAICT, there are two options to implement the other steps. The favourable one, if possible, would be to have the tests build without first building the program from source, i.e. running meson test without meson build. The program would be installed in the system and accessible normally via $PATH.
If that isn't enough, then I'd be fine with having to rebuild the entire thing. At that point, it's probably possible to combine the steps to only have to build it once, but the tests should still run and install independently as described above. This is most likely the option with the fewest changes.

Some context

I'm coming from the point of Arch Linux, which I personally use. As such, I'm quite familiar with the PKGBUILD way of packaging, where a check function is used for meson test. Ultimately, I'd like to see separate packages for testing that one can install after the fact, but for that to happen, the tooling must allow it via above changes. And once one tool has the feature, it's hopefully going to become widespread rather soon.
And I've chosen Meson as a starting place because I'm interested in it - of course- but also because I'm rather proficient in Python, so with some guidance I can help implement this change.

If I've missed some option or trick to make this possible, then please point it out, but I've not found anything with regards to this in the documentation.

Thank you for reading all this! Please give your thoughts on this for some discussion. And questions are always welcome!

[eli-schwartz]

Prior art: https://wiki.gnome.org/Initiatives/GnomeGoals/InstalledTests

Meson does "support" this by the way, inasmuch as it is mentioned at https://mesonbuild.com/Installing.html#installation-tags as something that, if it is part of a project, meson knows how to add install_tag defaults for it such that they are installed with meson install --tags tests.

[Vekhir]

The GnomeGoal has a slightly different angle, but the advantages are the same.

The installation tags can work, but only if there is support within the project. I don't see a reason why Meson can't automatically detect the tests, e.g.

e = executable('prog', 'testprog.c')
test('name of test', e)

Meson sees test(), the test has executable e, so e must be compiled and then included.

Of course, if you do stuff like in this discussion, then that's on you and you need to include the necessary files yourself. But the standard case should be automateable.

[eli-schwartz]

Note that installation tags don't actually cause more things to be installed than would otherwise be.

They simply take tests that the project has declared to be installed, and annotate them so that you can do partial component installs, which is sometimes useful for example for split packages.

I'm not sure how much sense it makes to automatically install any test() program that a project has (whether that projects expects it or not), unless the ultimate goal is to just ship the entire source code tree as part of the installed artifacts.

[Vekhir]

The test executables are compiled, so there is no source code necessary. Maybe the meson.build file might be necessary in some way, shape or form for the arguments and kwargs, but the tests itself don't need any code.

Projects, imo, should expect tests to be bundled. Of course, the current infrastructure doesn't really support and advertise it, so noone does, but that is no reason against change.
Some flexibility towards included tests is necessary. Whether that is a flag internal=[True|False] for test, where True means the test is not included, or an includelist/ignorelist at the level of meson install (with easy option to include/ignore all tests).

As a stepping stone, being able to run meson test without previously running meson build or meson compile on the whole source code (provided that the executables and libraries are installed system-wide, error otherwise) would increase the independence. This stepping stone would require access to the source code, but the ultimate goal remains to compile that too.

[eli-schwartz]

How would you handle tests that require test-data from the source tree? Either specified when building the test executable via c_args: ['-DTESTDIR=@0@'.format(meson.project_source_root() / 'test-fixtures')] or passed as a command-line argument in the test() function?

[Vekhir]

A new resource(...) function with accompanying object type might be the cleanest solution. In the simplest case, it would just take a path pointing to a file or directory.
E.g.:

test_resource = resource(meson.project_source_root() / 'test-fixtures')
test_exe = executable(..., c_args = ['-DTESTDIR=@0@'.format(test_resource.path())])
t = test(test_exe, resources = [test_resource])

This would also allow finer control over which data is copied, as data for tests that are not included can be ignored.
Note that the resources used must be declared via a new parameter resources which takes an array as shown above.

We can't enforce the use of the parameter right now, but the resource pattern is something we can document on the unit test doc page and give formal semantics to.