Testplan Design Patterns

Testplan Design Patterns

Table of Contents

• Organization of Tests
• Basic transaction
  • UseId
  • ExpectedStatus
  • AssignUuids
  • NoPatientId
  • Reusing Orchestration tests
  • Report/UseReport
  • Why are there multiple ways of communicating between test steps?
  • NoMetadata

Organization of Tests

Test, Section, Step

Basic transaction

The simplest testing role is to send a single transaction such as Provide and Register, we look at 11966/submit which is a Provide and Register to a Repository SUT. The test plan is:

<TestPlan>
  <Test>11966/submit</Test>
  <TestStep id="submit">
    <Goal>Transaction - Provide and Register to Repository under test
       Metadata pattern - Single Document submission
       Metadata contents - simple        Evaluation - Transaction is successful
    </Goal>
    <ExpectedStatus>Success</ExpectedStatus>
    <ProvideAndRegisterTransaction>
     <XDSb/>
     <MetadataFile>single_doc.xml</MetadataFile>
     <Document id="Document01">my_document.txt</Document>
    </ProvideAndRegisterTransaction>
  </TestStep>
</TestPlan>

<Test> identifies the test (as does the directory name holding test - they must be the same

<TestStep> is a Step which always holds a single transaction

<Goal> is documentation for the developer

<ExpectedStatus> indicates whether the transaction is expected to return an error.

<ProvideAndRegisterTransaction> tells us what transaction is sent. A TestStep must have a *Transaction element.

<XDSb/> indicates that the metadata template (listed under <MetadataFile/>) should be translated to XDS.b format. This is the default behavior now so this instruction is ignored. Originally when XDS.a and XDS.b were being tested this instruction allowed for the automatic transactions between version formats.

Now this TestStep is as simple as it gets but there is some automatic stuff going on behind the scenes. First the Orchestration phase almost always generates one or more Patient ID Reports. A Report is half of a Report/UseReport pair which is a Publish/Subscribe mechanism for sharing data between TestSteps. A TestStep reports a parameter/value pair. Another later TestStep uses the report. The UseReport statement identifies a Report of interest by specifying the Test/Section/Step/Name of the Report and assigning a local parameter name which can be coded in the Metadata template file.

Back to the Orchestration. It defines a Report named $patientid$ which is the default Patient ID for the test. This default is built into the processing of the metadata templates. If no further instructions are given, all Patient ID references will be updated to the value contained in $patientid$. Note that this only happens for ebRIM transactions with all the defaults in place. Other technologies will need a more manual approach. This automatic updating does not depend on the use of the $patientid$ variable. The test client knows the ebRIM structure and just does it. This simplistic mechanism really only works on the simplest tests. Usually a more advanced approach is needed where we specify exactly where the Patient ID comes from.

After the transaction is run various details are published to the log file. The automatically assigned IDs come first:

<AssignedPatientId>
   <Assign
         symbol="Document01"
         id="P1219090236.2^^^&1.3.6.1.4.1.21367.13.20.1000&ISO"/>
   <Assign
         symbol="SubmissionSet01"
         id="P1219090236.2^^^&1.3.6.1.4.1.21367.13.20.1000&ISO"/>
</AssignedPatientId>
<AssignedUids>
   <Assign
         symbol="Document01"
         id="1.2.42.20181219090315.2"/>
   <Assign
         symbol="SubmissionSet01"
         id="1.2.42.20181219090315.3"/>
</AssignedUids>

In this example output taken from 11966 we see the UID and Patient ID values that were automatically assigned. Each ebRIM object is assigned a new unique UID allowing the transaction template to be reused. If these values are needed in a later TestStep then the <UseId> instruction is used. Note that UUIDs were not assigned by default. Most transaction templates are codes with symbolic values instead of UUIDs again so the templates can be reused. In this example the symbolic ID was sent in the actual transaction allowing the Registry to assign a real UUID. We could have had the UUID assigned by the test client. That will be shown later.

The same content is published using the Report/UseReport mechanism. The UseId mechanism is much older than Report and not as flexible. So that follow-on tests can be coded using the newer UseReport we include the Reports in the logs as well.

<Reports>
    <Report
          name="Document01_uid">1.2.42.20181219090315.2</Report>
    <Report
          name="SubmissionSet01_uid">1.2.42.20181219090315.3</Report>
    <Report
          name="$patientid$">P1219090236.2^^^&1.3.6.1.4.1.21367.13.20.1000&ISO</Report>
 </Reports>

UseId

This instruction makes reference to an automatically published value from a previous test step. Continuing with the focus on test 11966 this Test Plan is the second section named eval

<TestPlan>
 <Test>11966/eval</Test>
 <TestStep
    id="eval">
    <Goal>Query the Registry simulator with a GetSubmissionSetandContents Stored Query       to verify the metadata containing a single DocumentEntry was forwarded in a       Register transaction.    </Goal>
    <ExpectedStatus>Success</ExpectedStatus>
    <StoredQueryTransaction>
       <XDSb/>
       <UseId
          testdir="../submit"
          id="SubmissionSet01"
          symbol="$ssuid$"
          step="submit"
          section="AssignedUids"/>
       <MetadataFile>query.xml</MetadataFile>
       <ExpectedContents>
          <SSwithOneDoc/>
          <DocApp/>
       </ExpectedContents>
    </StoredQueryTransaction>
 </TestStep>
</TestPlan>

The UseId references the submit section of this test (via filesystem relative reference) and identifies the test step. Here section refers to the Assigned* sections of the log (not the test section - bad choice of label). The reference is to the AssignedUids as opposed to AssignedPatientId. Symbol indicates the variable that has been coded into the metadata file. These variables are always code with the $ prefix and suffix so they can be found easily by our scanner. In this case the metadata file is the body of the query transaction. The id part of the UseId targets a particular ID published under AssignedUids.

ExpectedStatus

Test 11981/submit_bad_hash demonstrates the expectation of an error:

<TestPlan>
 <Test>11981/submit_bad_hash</Test>
 <TestStep
    id="submit">
    <ExpectedStatus>Failure</ExpectedStatus>
    <ExpectedErrorCode>XDSRepositoryMetadataError</ExpectedErrorCode>
    <ProvideAndRegisterTransaction>
       <XDSb/>
       <MetadataFile>single_doc.xml</MetadataFile>
       <Document
          id="Document01">my_document.txt</Document>
    </ProvideAndRegisterTransaction>
 </TestStep>
</TestPlan>

An expected status of Failure means that for the test to pass the transaction must fail. More specifically it must fail with the error code XDSRepositoryMetadataError.

AssignUuids

Test 11983/submit_no_doc uses the AssignUuids to make the test client translate the symbolic ids coded in the metadata template into real UUIDs (newly generated) before sending the transaction. The test plan is:

<TestPlan>
 <Test>11983/submit_no_doc</Test>
 <TestStep
    id="doc_missing">
    <ExpectedStatus>Failure</ExpectedStatus>
    <ExpectedErrorMessage>XDSMissingDocument</ExpectedErrorMessage>
    <ProvideAndRegisterTransaction>
       <XDSb/>
       <AssignUuids/>
       <MetadataFile>single_doc.xml</MetadataFile>
    </ProvideAndRegisterTransaction>
 </TestStep>
</TestPlan>

NoPatientId

Test 11986/submit has a hard-coded Patient ID that is to be sent so the automatic updating of the Patient ID from the Orchestration is turned off for this transaction:

<TestPlan>
 <Test>11986/submit</Test>
 <TestStep
    id="submit">
    <ExpectedStatus>Failure</ExpectedStatus>
    <ExpectedErrorMessage>XDSPatientIdDoesNotMatch</ExpectedErrorMessage>
    <ProvideAndRegisterTransaction>
       <XDSb/>
       <NoPatientId/>
       <MetadataFile>single_doc.xml</MetadataFile>
       <Document
          id="Document01">my_document.txt</Document>
    </ProvideAndRegisterTransaction>
 </TestStep>
</TestPlan>

Reusing Orchestration tests

Orchestration tests, the Test Plans used to create and submit Patient Ids and the test data that follows SHOULD NOT be reused. If say test 99999 is used in the Orchestration of actor X then it should not be reused in actor Y. If these actor types are both tested in the same Test Session then the test logs, our mechanism for preserving state about the running of tests, will overlap and something is going to break. If you need to reuse an real test or a step in orchestration then give it a new name.

Report/UseReport

Here we focus on the Registry tests specifically 11992/submit. In the Registry section of the Conformance Tool several tests are used to setup the environment of Patient Ids and test data. 11992/submit looks like

<TestPlan>
 <Test>11992/rplc</Test>
 <TestStep
    id="rplc">
    <Goal>Replace original DocumentEntry</Goal>
    <ExpectedStatus>Success</ExpectedStatus>
    <Rule>
       <Transaction>R</Transaction>
       <Assertion>Issue RPLC</Assertion>
    </Rule>
    <RegisterTransaction>
       <XDSb/>
       <NoPatientId/>
       <UseReport
          reportName="$patientid$"
          test="15817"
          section="pif"
          step="step"
          useAs="$patientid$"/>
       <UseReport
          reportName="docid"
          section="submit"
          step="submit"
          useAs="$rplc_doc_uuid$"/>
       <MetadataFile>metadata.xml</MetadataFile>
       <Report
          name="ssuid"
          section="InputMetadata">//*[local-name()='RegistryPackage']/*[local-name()='ExternalIdentifier'][@identificationScheme='urn:uuid:96fdda7c-d067-4183-912e-bf5ee74998a8']/@value</Report>
    </RegisterTransaction>
 </TestStep>
</TestPlan>

Note the use of the <NoPatientId/> instruction. This cancels the automatic Patient ID processing mentioned above and relies on specific linkage to set the proper Patient Id.

Seen is this Test Plan, the UseReport

<UseReport
        reportName="$patientid$"
        test="15817"
        section="pif"
        step="step"
        useAs="$patientid$"/>

uses the $patientid$ report out of test 15817/pif/step and applies it to the variable (useAs) $patientid$ in the metadata template. Test 15817 was run as part of Orchestration to establish a Patient in the Registry. In this case the variable $patientid$ must be specifically coded in the correct place in the metadata template. This UseReport matches up with

<TestPlan>
 <Test>15817/pif</Test>
 <Rule>Patient Identity Feed</Rule>
 <TestStep
    id="step">
    <Rule>
       <Transaction>Patient Identity Feed</Transaction>
    </Rule>
    <ExpectedStatus>Success</ExpectedStatus>
    <PatientIdentityFeedTransaction>
       <XDSb/>
       <NoMetadata/>
    </PatientIdentityFeedTransaction>
 </TestStep>
</TestPlan>

which you may note does not have an explicit Report. Transaction processing automatically publishes the Patient ID used in a Report named $patientid$. You know it was published because in the results displayed for 15817 it is shown.

Why are there multiple ways of communicating between test steps?

Basically, evolution (why do we have tailbones?). Originally the UseId mechanism was the only cross test communication device. It was tied to things automatically published and it would not work in the more general case. Also it was tied to using relative filesystem paths which are tough to get right and fail if you move something. Report/UseReport was created as a more flexible mechanism. But, there were so many tests already in the default test kit (several hundred) so that an easy path forward was needed so some automatic Reports were generated. Newer test all use plainly coded Report/UseReport. It would be wonderful to update all tests to use the latest mechanism but there has never been that opportunity.

NoMetadata

The above TestPlan uses the <NoMetadata/> instruction. The metadata file (there is one - it’s just generated automatically and not shown) has no ebRIM metadata so trying to parse and update it to XDS.b standards is not possible. NoMetadata is frequently used when processing non-ebRIM content for this reason.