HACKING.adoc

Running the testsuite

Useful Makefile targets

make parallel

runs the tests in parallel using the GNU parallel tool: tests run twice as fast with no difference in output order.

make all-foo, make parallel-foo

runs only the tests in the directories whose name starts with foo: parallel-typing, all-lib, etc.

make one DIR=tests/foo

runs only the tests in the directory tests/foo. This is often equivalent to cd tests/foo && make, but sometimes the latter breaks the test makefile if it contains fragile relative filesystem paths. Such errors should be fixed if you find them, but make one DIR=…​ is the more reliable option as it runs exactly as make all which is heavily tested.

make promote DIR=tests/foo

Most tests run a program and compare the result of the program, store in a file foo.result, with a reference output stored in foo.reference — the test fails if the two output differ. Sometimes a change in result is innocuous, it comes from an intended change in output instead of a regression. make promote copies the new result file into the reference file, making the test pass again. Whenever you use this rule please check carefully, using git diff, that the change really corresponds to an intended output difference, and not to a regression. You then need to commit the change to reference file, and your commit message should explain why the output changed.

Creating a new test

Dimensioning the tests

By default, tests should run well on small virtual machines (2 cores, 2 Gb RAM, 64 or 32 bits), taking at most one minute, and preferably less than 10 seconds, to run on such a machine.

Some machines used for continuous integration are more capable than that. They use the OCAML_TEST_SIZE environment variable to report the available resources:

OCAML_TEST_SIZE	Resources	Word size
1 or unset	2 cores, 2 Gb RAM	32 or 64 bits
2	4 cores, 4 Gb RAM	64 bits
3	8 cores, 8 Gb RAM	64 bits