6 Testing and modelling using coded rules
The concept of ‘test suites’ arose in the software development industry.
In the context of coded rules, a ‘test’ is:
- a description of a scenario with given inputs, and
- what the expected output(s) should be.
That is, what we are seeking to test is whether your rules work as you intended - that the coded rules and the system preserve the policy intent of the rules.
If rulesets are coded, this opens up the opportunity to automate testing, or create tools to make testing easier.
With respect to automated testing, depending on the kind of ruleset, it is possible to generate a suite of tests - every possible scenario and every possible correct outcome, and then run all scenarios to see if there are bugs or flaws in the rules or the system. Both the generation of the test suite and the running of tests can be automated. This allows us to not only quickly find flaws, but to demonstrate the validity and accuracy of the coded rules.
On 2 February 2021, Austlii coded the Treasury Laws Amendment (News Media and Digital Platforms Mandatory Bargaining Code) Bill 2020. They used this to create a tool using their DataLex platform to let users apply and test the draft code. In particular, the tool helps users determine if a media organisation has standing to participate in the bargaining process set out in the code.
If your ruleset is not too complex, the policy owner or SME can write the tests - setting out each possible scenario, the inputs, and the correct outcomes given those inputs. This can be done in a spreadsheet format, having regard to the possible inputs.
For example, if you have a fuel rebate where an application is eligible if:
- they are over 18 years of age AND
- live in a rural area AND
- and own a car then the test suite might look as follows:
If you have a complex calculation - such as a calculation that depends on numerous variables - writing tests manually may become time consuming. You should consider automating the creation of your test suite
TJ Harrop has built an an OpenFisca test generator, which can be used to quickly generate test suits for OpenFisca by running through all permutations of the variables. You may wish to name each test manually for ease of reference.
[tba]
A rules as code approach offers the opportunity to conduct more efficient policy modelling.
Once the rule base has been coded, and you have developed an interface, you can quickly and efficiently model different outcomes by just modifying the rules and considering the resulting impacts.
Mes Aides: A Rules as Code-enabled benefits and entitlements simulator, originally developed by the French Government and now run as a community project.
Regan Meloche at Code for Canada has been experimenting with Rules as Code-enabled policy modelling: Introducing the Policy Difference Engine 28 November 2020