Creating custom Conformance Tool test collection

Creating a custom Conformance Tool

Some users want to alter the tests that are presented by adding tests, changing the tab that a test appears under, or removing a test from the display. This page describes some strategies.

Background

The Actor types that are configured into the Conformance Tool are build strictly in Java code and take considerable work to alter. The work is in the design and implementation of the orchestration - the initialization of the test environment for a particular profile/actor combination through the launching/configuration of a simulator and the loading of simulators with initial data. This requires custom code on both the UI and server side and requires that an Actor simulator exists.

Tests are organized by profile/actor/option. Options can be formal options defined by IHE Profile or simple groupings that are created for organizing tests.

There are a bunch of key things to understand to create your own organization of tests for a Profile/Actor combination.

1. How option tabs are created - option tabs are the third row of tabbing on the Conformance tool display. They are nested inside the Actor and Profile tab layers.
2. How tests are associated with an actor/profile/option.
3. Are the tests being organized native to Toolkit (contained in the WAR) or are they private (stored in the External Cache) or a combination of both? The answer to this question will determine how hard it is to maintain the test organization.

Configuring option tabs

Tests are defined into one or more Testkits. One Testkit is delivered with Toolkit while the rest are configured locally in the External Cache.

Each test definition is a directory where the test name is the directory name. Within the directory is a file collections.txt. This file contains a collection of test labels (if this were a database they would be called keys), one per line. These labels can associate a test with a particular actor/profile/option as discussed in Test collections and Conformance tool tabs.

A configuration file

toolkitx/tool-tab-configs/ConfTestsTabs.xml

specifies the tab organization in the Conformance tool. This is documented in Test collections and Conformance tool tabs.

Controlling the tests that appear in a tab

When a tab is specified in ConfTestsTabs.xml it does so with a nested configuration of actor, profile, option. These are present in the type attribute of the nested tab and tabs elements. The tests that are associated with an actor/profile/option combination by codes in the individual collections.txt files found in the test definitions.

The syntax for the labeling is

actor(profile)_option

as documented in Test collections and Conformance tool tabs. So if a test (in its collections.txt file) is labeled with reg(xds)_mu and those three types are properly nested in ConfTestsTabs.xml then that test will appear under that tab.

To reorganize internal tests or organize external tests the effort goes into modifying the collections.txt files of the tests of interest and altering ConfTestsTabs.xml.

Reorganizing internal tests

This is a bit tricky. Not technically but maintenance-wise. You are going to alter the collections.txt file of multiple tests and the system wide ConfTestsTabs.xml file. ConfTestsTabs.xml is relatively easy to maintain. Create a custom copy of the file and replace the delivered version in the WAR file before deploying. This is only easy if the new Toolkit version has not updated this file as well. You are responsible for any merging necessary. If you find it necessary to alter collections.txt files then how are you going to maintain your version. I see two approaches and both have problems.

First you could replicate the affected tests in a private Testkit in the External Cache. Even though you only want to alter collections.txt you must replicate the entire test definition. When a new Toolkit release is installed how do you handle your updates? Does the core Toolkit have updates/fixes for any of the tests you have replicated in the External Cache? How do you merge the updates? You need a strategy.

The alternative is to only replicate the various collections.txt files and applying this update to the Testkit in the WAR. I would do this by creating a file hierarchy replicating the Testkit structure but only filling in the collections.txt files that you have modified. To update a new release you can do a recursive copy. This could still conflict with the updates in the Toolkit release but it is less likely.

You need to understand what you are updating to pick a strategy here.

Organizing external tests

This is a simpler case where ALL the tests being used are custom to your project and none of the Toolkit delivered tests are being used. This would mean that you have a custom Testkit already constructed in the External Cache and possibly an updated ConfTestsTabs.xml file. You would replace ConfTestsTabs.xml, delete the internal Testkit and run with only your custom Testkit.

When does test organization happen

ConfTestsTabs.xml and the various collections.txt files are scanned when the WAR/servlet/Toolkit is started. There are two separate tasks run. First a consistency checker is run on these files. Does ConfTestsTabs.xml refer to a actor/profile/option that doesn't exist? Second the tabs are build based on these files.