Integration of Orchestration and new Conformance Test tool

There are several areas of development that are coming together to support Conformance testing. This article discusses the use cases and how these technologies can be integrated.

Use cases

There a collection of use cases driving this work:

• New UI for conformance testing
• This UI is state-full - you can review logs in the tool that runs the tests
• Many tests require some form of orchestration
• Ability to launch a specific test against a specific SUT from a URL.

What is orchestration

The definition we started with half a year ago when developing the XCA test suite was a UI driven script that composed simulators and preloaded them with test data. This was used to establish a test environment for an SUT that required a complex testing environment. We have successfully built tests for XCA and On-Demand in this way.

The design of the new conformance test tool has established the need for an expanded definition of Test Session. Currently it is used for two things: container for simulators and container for test results. This expanded definition brings in the linkage between a test session and an environment (cert and codes) and site. The environment linkage is needed because the test kit is linked to the codes - tests can be updated to use a specified codes definition. The site linkage is called for because producing a set of test results run across multiple sites is meaningless - some of these tests are run against my development system and other against my product. Totally meaningless.

I now wish to extend this definition of orchestration to include some of the use cases listed above, specifically the state-full UI and launch from URL.

The above description of orchestration is presented from the point of view of processing. This new definition looks at orchestration from the view of linkage of one test to the results of another test. The processing view says that before I run this test I must run this script to establish the test environment. The new linkage view says this test is dependent on a collection of state elements: test logs and simulators. If this state does not exist then create it before running the test.