Conformance Test tool developer guide

This guide describes how to develop tests for the Conformance Test tool.

Test development and management is organized into two levels. The low level is the writing of test plans, detailed scripts for exercising the System Under Test. The upper level is the organizing of these test plans into a testing environment. This upper level is the focus of the Conformance Test tool.

Test plans are grouped into test collections - simple ordered lists of tests. Within the Conformance tool there is a hierarchical organization:

Conformance tool
  Actor type
    Required and optional test collections

Each Actor type is supported by an orchestration module that creates the testing environment need to support the included tests. In the future each test collection could have its own orchestration module but this approach has not yet been used.

Orchestration refers to the creation of the environment that supports a collection of tests. Orchestration can include building and configuring supporting simulators and loading simulators or the system under test with data.

Orchestration

Orchestration is a process with several distinct steps.

Establish Test Context

Initialize Testing Environment

User configuration of System Under Test

How tests are organized to fit into the Conformance tool

Tests are added to the Conformance tool by their test collection definition.

Each test, optionally, has a file named collections.txt which lists the collections the test belongs to. This is the primary indexing of tests and a test without such an index will not be available in toolkit.

Collections.txt is a simple list of collection names, one per line. For example test 12313 has in collections.txt

rg

indicating that it is a test for the Responding Gateway actor. Notice that this code is the one defined in ActorType.java. This index will cause this test to be listed on the Responding Gateway portion of the Conformance tool under the Required Tests tab.

Compare this with the collections.txt file for 12312_Xuaunsigned test:

rg_xua

This collection name has two parts, rg and xua. The first part says that it is a test for the Responding Gateway actor. The second part says that it is not a required test but instead a test for an option, specifically the xua option. This will show up on the correct option tab within the Conformance tool.

The configuration for Conformance tool actor options is found in the class

The configuration of options for actors with the Conformance tool uses two levels of specification. The first is in the class ActorOptionManager which has the table with entries like:

actorOptions.put("rg", BuildRGTestOrchestrationButton.ACTOR_OPTIONS)

This says that the Responding Gateway actor (code rg) is supported by the options table at BuildRGTestOrchestrationButton.ACTOR_OPTIONS which looks like:

new ActorAndOption("rg", XUA_OPTION, "XUA Option", false))

Note that there are two problems with this. First "rg" should be replaced by a reference to ActorType and this configuration should be in a more central location. Both of these deficiencies will be address in a future release of toolkit.

For now…​

XUA_OPTION is a static reference to the string "xua" which was the second (option) component of the entry in collections.txt.

"XUA Option" is the name to be displayed on the tab in the UI.

false indicates that the tests for this option can be started by the Conformance tool (some tests must be initiated by the SUT instead).