CI Build

The ofn-install Travis build runs through some of the core playbooks and performs some additional tests.

When we run a playbook in the build environment, each Ansible task is effectively a unit test in itself. It does a specific thing, and will either pass or fail. For the most part we don't need to do much more, but there have been a couple of cases where playbooks have been running without failing but not actually doing what they were intended to do.

With this in mind, we can perform a few basic sanity checks and smoke tests with this suite.

The test suite

The test suite is directed from a central file called suite.yml in the /tests folder. This file will run a number of playbooks from the repo, and can perform some additional tests. For each playbook tested, we:

• Optionally include some test setup
• Run the playbook
• Perform some test(s)

Test files are grouped in folders corresponding to the names of the playbooks being tested. So for the rollback playbook we have this structure:

/tests
   /rollback
      setup.yml
      test.yml
  

And in suite.yml for testing the rollback playbook, we have this entry:

# Perform a rollback
- import_playbook: ../tests/rollback/setup.yml
- import_playbook: ../playbooks/rollback.yml
- import_playbook: ../tests/rollback/test.yml

Fresh install vs live instance

The first thing we do in the test suite is run site.yml. This is a special playbook that runs setup.yml, provision.yml, and deploy.yml. This takes the target from a fresh Ubuntu install to a fully set-up OFN instance.

As the environment is very different during the first run on a fresh intall and subsequent runs on a live instance, we check deploy.yml in both scenarios by running it again after site.yml.

Naming of test and setup tasks

In order to give clear feedback in the output of the tests, the Ansible tasks in setup.yml and test.yml files use the following naming pattern: <type> | <playbook-name> | <description>

So for example, in tests/rollback/setup.yml we have a task named: setup | rollback | set a file in app folder, and in tests/rollback/test.yml we have a task named: test | rollback | expect files in app folder to be reverted.

Setup tasks should clearly describe what they are doing, and test tasks should clearly describe what they expect so the output is easy to read if they fail.

Limitations

Currently we skip a couple of things that are difficult to test in a non-live server. These include:

• Certbot TLS certificate installation
• Notifications to Slack, Bugsnag etc