Test Documentation

Writing test documentation for XDS Toolkit

This page is obsolete. Please see Writing-Conformance-Tests for the current stuff.

The documentation style documented here is meant to coordinate with the new Conformance Test Tool.

Test structure

A test is made up of sections which contain one or more steps. The following rules apply:

• A step corresponds to a single transaction.
• A section contains 1 or more steps.
• A section can be run independently. When run, it executes all its steps in order specified.
• The common reason to pack two steps into a section is to make sure they are always executed together. An example is a Patient Identity Feed transaction followed by a Register Document transaction. Every time you register this document (in our example scenario) you want to assure it is given a new/unique Patient ID. To ensure this the two transactions are packed into one section (each transaction in its own step).
• There are older tests where independent operations are packed into steps in a section. This is discouraged.
• In a multi-section test, the sections can be run independently, unless...
• Sections of a test can have dependencies with one section needing the output of a previous section. Actually it is the steps that have dependencies but given the one step per section rule...
• Sections run in a proscribed order.
• When a test is run, all of it sections are run in order.

Test directory structure

A part of toolkit called the testkit contains the test definitions. A test is a directory with proscribed content. An example:

12360/
    readme.txt
    index.idx
    collections.txt
    submit/
        testplan.xml
        single_doc.xml
        my_document.xml
    query/
        testplan.xml
        query.xml
    retrieve/
        testplan.xml
        query.xml

The directory 12360 is a test definition. Its ID is 12121.

readme.txt - contains two things. The first line is the short description of the test. The rest of the file is the long description of the test.

sections - the directories submit and query are sections. They are identified as sections because they are directories and they are listed in index.idx.

index.idx - list of section names in order of execution. Each section name is placed alone on a line. In this example the contents are:

submit
query
retrieve

Note that directories within a test definition that are not listed in index.idx are not sections. They are ignored.

section structure - each section has a testplan.xml that controls the execution of the section. There may be other files in the section directory that are reference from the testplan.xml file.

A testplan looks like

<?xml version="1.0" encoding="UTF-8"?><TestPlan>
<Test>12360/query</Test>
  <Rule>Retrieve metadata to get access to XDSDocumentEntry.repositoryUniqueId and XDSDocumentEntry.uniqueId</Rule>
  <TestStep id="by_uid">
    <Goal>
      Transaction - GetDocuments Stored Query
      Target - DocumentEntry just submitted
      Evaluation - Transaction is successful
    </Goal>
    <ExpectedStatus>Success</ExpectedStatus>
    <StoredQueryTransaction>
      <XDSb/>
      <UseId testdir="../submit" id="Document01" symbol="$docuid$" step="submit" section="AssignedUids"/>
      <MetadataFile>query.xml</MetadataFile>
      <Report name="repuid" section="Result">
            //*[local-name()='ExtrinsicObject']/*[local-name()='Slot'][@name='repositoryUniqueId']/*[local-name()='ValueList']/*[local-name()='Value']
      </Report>
      <ExpectedContents>
        <Documents count="1"/>
      </ExpectedContents>
    </StoredQueryTransaction>
  </TestStep>
</TestPlan>

Test documentation

Tests and steps have documentation. Sections are just groups of steps so they have no documentation.

Test documentation is placed in the readme.txt file. The first line is a summary and the rest of the file is the details. This should describe the overall goals of the test and the environment the test runs in. It should not describe the sections/steps of the test.

Test documentation can be written as plain text, HTML, or a subset of Markdown formats. Note that the file extension does not reflect the format. The format is detected based on the content.

Step documentation is placed in the testplan. A testplan contains one or more elements. Each TestStep should have a element as shown above. This Goal element contains the documentation of the test. The Goal element contains pre-formatted text. This content is presented exactly as entered without HTML or Markdown formatting.

The test documentation comes from readme.txt:

# Repository handling of mimeType

## Overview

Verify that the Repository accepts, stores, and returns the mimeType present
in the Provide and Register transaction.

Before running, configure Repository to forward Register.b transactions to the Public
Registry.

This is using Markdown syntax. This syntax should be used for all tests because later a consolidated set of documentation for all tests will be generated and the formatting should be consistent.

The Test short description in the first line should be formatted as H1 (single # sign). Next should be a H2 titled overview. What follows is the details of the documentation.

The step documentation is shown in the example.

Step documentation contents

The two basic kinds of steps/transaction are submit and query. For submit type transactions I suggest the following items in the documentation:

Transaction - Provide and Register (for example)

Metadata pattern - Single Document submission (for example)

Metadata contents - The phony mimeType text/goofy (what is special or important about the metadata)

Evaluation - Transaction is successful (how is the step evaluated for success)