Forall .

Property-based testing for Ruby (adapted from Jacob Stanley's Hedgehog library for Haskell, and an older project I made named Propr).

Introduction

The usual approach to testing software is to describe a set of test inputs and their expected corresponding outputs. The program is run with these inputs, and the actual outputs are compared to what's expected to ensure the program behaves correctly. This methodology is simple to implement and automate, but has some problems like:

• Writing test cases is tedious and repetitive.
• Only edge cases that occur to the author are tested.
• It can be difficult to see which parts of the test input are mere prerequisites rather than essential.
• Getting 100% code coverage with trivial tests doesn't offer much assurance.

Property-based testing is an alternative and complementary approach in which the binary relations between attributes of inputs and desired output are expressed as functions, rather than enumerating particular inputs and outputs. The properties specify things like, "assuming the program is correct, when its run with any valid inputs, the inputs and the program output are related by f(input, output)".

Properties

The following example demonstrates testing a property with a specific input, then generalizing the test for any input.

describe Array do
  include Forall::RSpecHelpers

  describe "#+(other)" do
    # Traditional unit test
    it "sums lengths" do
      xs = [100, 200, 300]
      ys = [400, 500]
      expect((xs + ys).length).to eq(xs.length + ys.length)
    end

    # Property-based test
    it "sums lengths" do
      ints = random.array(random.integer(0..999))

      forall(random.sequence(ints, ints)) do |xs, ys|
        (xs + ys).length == xs.length + ys.length
      end
    end
    # property("sums lengths"){|xs, ys| (xs + ys).length == xs.length + ys.length }
    #   .check([100, 200, 300], [500, 200])
    #   .check{ sequence [Array.random { Integer.random }, Array.random { Integer.random }] }
  end
end

The following example is similar, but contains an error in the specification

describe Array do
  include Propr::RSpec

  describe "#|(other)" do
    # Traditional unit test
    it "sums lengths" do
      xs = [100, 200, 300]
      ys = [400, 500]

      # This passes
      expect((xs | ys).length).to eq(xs.length + ys.length)
    end

    # Property-based test
    it "sums lengths" do
      ints = random.array(random.integer(0..999))

      forall(random.sequence(ints, ints)) do |xs, ys|
        (xs | ys).length == xs.length + ys.length
      end
    end

    # property("sums lengths"){|xs, ys| (xs | ys).length == xs.length + ys.length }
    #   .check([100, 200, 300], [400, 500])
    #   .check{ sequence [Array.random{Integer.random(min:0, max:50)}]*2 }
  end
end

When this specification is executed, the following error is reported.

$ rake spec
..F

Failures:

  1) Array#| sums lengths
     Failure/Error: raise Falsifiable.new(counterex, m.shrink(counterex), passed, skipped)
     Propr::Falsifiable:
       input: [], [0, 0]
       after: 49 passed, 0 skipped
     # ./lib/propr/rspec.rb:29:in `block in check'

Finished in 0.22829 seconds
3 examples, 1 failure

You may have figured out the error is that | removes duplicate elements from the result. We might not have caught the mistake by writing individual test cases. The output indicates Forall generated 49 sets of input before finding one that failed.