property-based-testing.md

Property-Based Testing

We briefly mentioned property checks when talking about design-by-contracts. There, we used the properties in assertions. We can also use the properties for test case generation instead of just assertions.

Given that these properties should always hold, we can test them with any input that we like. We typically make use of a generator to try a high number of different inputs, without the need of writing them all ourselves.

These generators often create a series of random input values for a test function. The test function then checks if the property holds using an assertion. For each of the generated input values, this assertion is checked. If we find an input value that makes the assertion to fail, we can affirm that the property does not hold.

The first implementation of this idea was called QuickCheck and was originally developed for Haskell. Nowadays, most languages have an implementation of QuickCheck, including Java. The Java implementation we are going to use is made by Paul Holser and is available on his GitHub. All implementations of QuickCheck follow the same idea, but we will focus on the Java implementation.

Getting started with property-based tests

How does it work?

• First, we define properties. Similar to defining test methods, we use an annotation on a method with an assertion to define a property: @Property. QuickCheck includes a number of generators for various types. For example, Strings, Integers, Lists, Dates, etc.

• To generate values, we add some parameters to the annotated method. The arguments for these parameters will then be automatically generated by QuickCheck. Note that the existing generators are often not enough when we want to test one of our own classes; in these cases, we can create a custom generator which generates random values for this class.

• QuickCheck handles the number of generated inputs. After all, generating random values for the test input is tricky: the generator might create too much data to efficiently handle while testing.

• Finally, as soon as QuickCheck finds a value that breaks the property, it starts the shrinking process. Using random input values can result in very large inputs. For example lists that are very long or strings with a lot of characters. These inputs can be very hard to debug. Smaller inputs are preferable when it comes to testing. When an input makes the property fail, QuickCheck tries to find shrink this input while it still makes the property fail. That way it gets the small part of the larger input that actually causes the problem.

As an example: a property of Strings is that if we add two strings together, the length of the result should be the same as the sum of the lengths of the two strings summed. We can use property-based testing and the QuickCheck's implementation to make tests for this property.

@Runwith(JUnitQuickcheck.class)
public class PropertyTest {

  @Property
  public void concatenationLength(String s1, String s2) {
    assertEquals(s1.length() + s2.length(), (s1 + s2).length())
  }
}

concatenationLength had the Property annotation, so QuickCheck will generate random values for s1 and s2 and execute the test with those values.

Property-based testing changes the way we automate our tests. We have only been automating the execution of our tests; the design and instantiation of test cases were always done by us, testers. With property-based testing, by means of QuickCheck's implementation, we also automatically generate the inputs of the tests.

Property-based testing and AI

A lot of today's research goes into a AI for software testing is about generating good input values for the tests. We try to apply artificial intelligence to find inputs that exercise an important parts of the system.

While the research's results are very promising, there still exist difficulties with this test approach. The main problem is that if the AI generates random inputs, how do we know for sure that the outcome is correct, i.e., how do we know that the problem behaved correctly for that random value? With the unit test we made so far we manually took certain inputs, thought about the correct outcome, and made the assertion to expect that outcome. When generating random inputs, we cannot think about the outcome every time. The pre- and postconditions, contracts, and properties we discussed in this chapter can help us solve the problem. By well-defining the system in these terms, we can use them as oracles in the test cases. The properties always have to be true, so we can use them in all the randomly generated test cases.

We will discuss more about AI techniques in a future chapter.

{% set video_id = "7kB6JaSH9p8" %} {% include "/includes/youtube.md" %}

Exercises