Acceptance Testing

Introduction

After installation and configuration of XDSTools in a tomcat environment, these are tests to run to check the configuration.

Port Tests

Web Server Port

Tomcat is configured to run at some port. The default port is 8080. The toolkit properties as an entry for that port number that needs to match the tomcat configuration. In addition, any firewall settings need to allow external systems to connect to that port.

1. Make sure the tomcat port and xdstools configuration port are using the same value. In some settings, the tomcat port is obscured by an Apache front end/redirect. In order to check the tomcat port, you can point your web browser directly to the proper port. For example, this could be: http://host:8080/xdstools4.
   • If you are unsure if tomcat is listening on a particular port, use telnet on the server itself: telnet localhost 8080. If telnet makes a connection, you at least know that something is listening on that port.
2. Make sure that external systems can reach the specified port. This means you are checking both local firewall rules on the server and firewall rules established by the organization hosting the server. The best check for this is to use a web browser outside the server's network (i.e., from your own office) and connect to http://host:port/xdstools4. If this does not work, try the telnet trick listed above.

If you are not able to connect directly to the published port, this might be a security feature implemented by the organization hosting the tomcat server. Direct access to that port might allow someone to connect to http://host:port which would give them access to the tomcat console. The organization might only open host:port to specific IP addresses that are vetted for testing and not from your system.

HL7 Listener Port

Each Document Registry simulator has an HL7 V2 listener. Each listener is created on a separate port. Port numbers are configured in the Toolkit Properties.

1. Use the web user interface to create a Test Session (say hl7).
2. In your Test Session, create a Document Registry simulator; use the value "reg" for Simulator ID.
3. Open the configuration for the new Registry simulator and make a note of the port for the HL7 listener.
4. Use telnet to connect to that port from the server itself (see syntax above). This will ensure that the HL7 listeners are started and the port is ready to accept a connection.

There should be no reason for the XDSTools software to receive an external HL7 V2 feed. There are no tests for Patient Identifier Sources. Therefore, there is no need to test the connection to the HL7 V2 listener from outside the server's firewall environment.

Conformance Tests Runs

Document Registry

1. Create a Test Session (alpha).
2. In that Test Session, create a Document Registry simulator. Use the value "sim-rr" for Simulator ID. The simulator name will be "alpha__sim-rr".
3. Select the Conformance Tests control and then Document Registry from the pane that is displayed.
4. Inside the Document Registry pane, activate the button to initialize the test environment.
5. Open the Test Context box. Make sure the proper Test Session (alpha) is selected. For the System Under test, select the Registry simulator you created above (alpha__sim-rr).
   • I have had some issues with the tools just after this step. I recommend you close the ConfTests tab at this point and then select the Conformance Tests control again.
6. Run the first Document Registry conformance test. That test should complete successfully and the test line turn green.
7. Assuming the step above worked, activate the button to run all of the tests. Each test will run in succession and the status should be green for each one.
8. Cleanup after the tests are completed:
   • Delete the registry simulator you created (alpha__sim-rr). You should use the Simulator Manager in the toolkit.
   • Delete the test logs for the registry tests. You will have to do this from the command line on the VM. Look in the TestLogCache folder in the XDSTools external cache. There will be a folder with the name of the test session you created (alpha). Remove that folder and everything under it. Note that the XDSTools software may create an empty session folder (alpha) because it has that context. To really remove that folder, you will need to stop tomcat, remove the folder, and restart tomcat.

Document Repository

1. Create a Test Session (beta).
2. In that Test Session, create a Document Repository simulator. Use the value "sim-rep" for Simulator ID. The simulator name will be "beta__sim-rep".
3. Select the Conformance Tests control and then Document Repository from the pane that is displayed.
   • The system will open a window to ask you to enter Test Context. Select the repository simulator you created as the system under test. Make sure the proper test session is selected.
4. Inside the Document Repository pane, activate the button to initialize the test environment. Make sure you have the Reset option checked. This will direct the tools to send ADT messages to the supporting registry simulator. These ADT messages are needed for the test procedure.
5. Configure the Document Repository simulator you created to use the Document Registry simulator created when you initialized the test environment.
   • In the simulator tab, edit the Document Registry simulator and copy the HTTP URL for the Register endpoint.
   • Edit the Document Repository simulator; paste the value you just copied into the Register endpoint for this repository simulator.
6. Run the first Document Repository conformance test. That test should complete successfully and the test line turn green.
7. Assuming the step above worked, activate the button to run all of the tests. Each test will run in succession and the status should be green for each one.
8. Cleanup after the tests are completed:
   • Delete the repository simulator you created (beta__sim-rep). You should use the Simulator Manager in the toolkit.
   • Delete the test logs for the registry tests. You will have to do this from the command line on the VM. Look in the TestLogCache folder in the XDSTools external cache. There will be a folder with the name of the test session you created (beta). Remove that folder and everything under it. Note that the XDSTools software may create an empty session folder (beta) because it has that context. To really remove that folder, you will need to stop tomcat, remove the folder, and restart tomcat.

Initiating Gateway

1. Create a Test Session (gamma).

Responding Gateway

1. Create a Test Session (delta).

Imaging Document Consumer (theta)

1. Create a Test Session (theta).
   • Do not create any simulators by hand.
   • Do create a system using the xdstools software. Call it theta_idc_sut.
2. Select the Conformance Tests control. Do not open the Imaging Document Consumer tab yet.
3. Open the Test Context box (upper right). Make sure the proper test session (theta) is selected. More importantly, select theta_idc_sut as the system under test. Click on the Assign button and then close the window.
4. Open the Imaging Document Consumer tab. This action will direct the software to automatically create the orchestrated environment, including send KOS objects to the Document Repository/Registry simulator.
5. You will now simulate the Imaging Document Consumer performing retrieve functions.
6. Perform queries needed for test 4831.
   • Select "Find Documents" under Queries & Retrieves.
   • In that pane, select the Document Registry that was created as part of the environment (theta_rr).
   • Paste this value in for patient ID: IDCAD001-a^^^&1.3.6.1.4.1.21367.2005.13.20.1000&ISO
   • Activate the Run button. That will send a stored query with this patient identifier.
   • Select Inspect Results after the query is complete. That will open the Inspector pane for this query.
   • Select "Action: Get Full Metadata for next 1". If this shows more than one document, an error has occurred during the orchestration step. There should be a single KOS object for this patient ID.
   • Expand "Documents" and then "DocumentEntry" and then retrieve the document (KOS).

Imaging Document Source (epsilon)

These instructions assume you know where to place a folder in the external cache to define a new test or override an existing test. We are adding a new test for the Imaging Document Source that simulates the system under test by sending Provide and Register transactions with KOS objects to the Document Repository.

1. This first step will require you to restart the Tomcat server that is hosting the XDSTools software. This is needed to signal the toolkit to read a folder with repository data that simulates the Imaging Document Source sending KOS objects to the repository simulator.
   • The file ids_2018-4800-regression.zip one folder (ids_2018-4800-regression) with sub folders and files to execute a conformance test for an Imaging Document Source.
   • Find the folder environment/default/testkits/default/tests in the external cache. This folder allows you add/overwrite conformance tests. Unzip ids_2018-4800-regression.zip in this folder to create the new folder environment/default/testkits/default/tests/ids_2018-4800-regression.
   • There are three subfolders that each contain a testplan.xml file. That testplan.xml file hardcodes the URL to submit data to the Document Repository as follows: http://localhost:9280/toolkit/sim/epsilon__rr/rep/prb. Assuming that you used the session epsilon as directed, you will only need to review/change the port number (9280) and first part of the path in the URL (toolkit) to match your environment. Make this correction for all three transactions.
   • Stop/restart the Tomcat server. The toolkit will locate this folder add it to the set of conformance tests for the Imaging Document Source actor.
2. Create a Test Session (epsilon).
   • Do not create any simulators by hand.
3. Select the Conformance Tests control and then Imaging Document Source from the pane that is displayed.
4. Inside the Initiating Imaging Document Source pane, activate the button to initialize the test environment.
   • The system will open a window to ask you to enter Test Context. Close that window without selecting anything. You will select context later.
5. Open the Test Context box. Make sure the proper Test Session (epsilon) is selected.
   • When the testing environment is created, an IDS simulator is automatically created. The name is $session__simulator_ids.
   • For the System Under test, select the IDS simulator that was automatically created (e.g., epsilon__simulator_ids).
   • Close the ConfTests tab.
6. You should have closed the Conformance Tests tab in one of the steps above. Select that function again (Conformance Tests in the lower left flap) and then select Imaging Document Source.
7. You now have to simulate having an Imaging Document Source submit KOS objects for three separate imaging studies. This is where you will use the files extracted into the external cache:
   • Execute test ids_2018-4800-regression. This sends three Provide and Register transactions to the Document Repository. Each transaction contains one KOS object.
   • If this step fails, the first place to look is a the three testplan.xml files under this test. Make sure the URL for the Document Repository is correct.
8. Run Imaging Document Source conformance test ids_2018-4801a. It evaluates the KOS object and metadata for the first imaging study and should succeed.
9. Assuming the test above runs correctly, run tests ids_2018-4801b and ids_2018-4801c in the same fashion and examine the results. You should see no errors.
10. Assuming the step above worked, activate the button to run all of the tests. Each test will run in succession, including the setup test for regression testing.
11. After completing these tests, you should remove test ids_2018-4800-regression. If this test remains in the system, it will be exposed to organizations testing the Imaging Document Source actor and will be confusing. It may also give them the opportunity to use this test to send or review the metadata that they are responsible to create.
    • After you have removed the ids_2018-4800-regression from the external cache, you will need to restart the tomcat server.

Imaging Document Consumer and Imaging Document Source

There are extra tests of the Imaging Document Source simulator in the context of both IDS and IDC testing that are not handled well by manual button pushing. These tests were introduced in March of 2019 to make sure the Imaging Document Source simulator could respond to requests for JPEG files. These extra tests are both a test of the software changes as well as you having the proper JPEG files in place in the external cache.

After you have completed the manual acceptance testing for the Imaging Document Consumer and Imaging Document Source tests, you will have test sessions epsilon and theta with appropriate simulators. The Toolkit software distribution has a folder: bin/src/main/imaging-regression

Using a terminal emulator:

• cd bin/src/main/imaging-regression
• ./imaging_regression.sh cache toolkit_URL IDS_session IDC_session

where

• cache is the folder of the external cache. The external cache will contain the folder ImageCache. Do not point to ImageCache directly.
• toolkit_URL is just that. It will be something like http://localhost:8080/toolkit
• IDS_session is the test session you created for the IDS tests. This value should be epsilon.
• IDC_session is the test session you created for the IDC tests. This value should be theta.

The tests execute in three phases. The first phase looks through the images in the external cache for the Imaging Document Source tests and makes certain that all files (DICOM and JPEG) can be retrieved. You should make sure that 47 files are processed successfully.

The second phase looks through the images for the Imaging Document Consumer tests and makes certain that all files can be retrieved. You should make certain that 257 files are processed successfully.

The third phase executes for exception cases (#101, 102, 103, 104). These are tests of the Imaging Document Source simulator where the client application has sent the wrong parameter or has excluded a required parameter. Each of these cases should be executed with no error messages.

Exception case number: 101, Description: Request image/jpeg where no representation exists; expect 404 error

Exception case number: 102, Description: Omit object UID; expect 400 error

Exception case number: 103, Description: Omit series UID; expect 400 error

Exception case number: 104, Description: Omit study UID; expect 400 error

Initiating Imaging Gateway

1. Create a Test Session (zeta).
   • Do not create any simulators by hand.
2. Select the Conformance Tests control and then Initiating Imaging Gateway from the pane that is displayed.
3. Inside the Initiating Imaging Gateway pane, activate the button to initialize the test environment.
   • The system will open a window to ask you to enter Test Context. Close that window without selecting anything. You will select context later.
4. Open the Test Context box. Make sure the proper Test Session (zeta) is selected.
   • When the testing environment is created, an IIG simulator is automatically created. The name is $session__simulator-iig.
   • For the System Under test, select the IIG simulator that was automatically created (e.g., zeta__simulator-iig).
   • I have had some issues with the tools just after this step. I recommend you close the ConfTests tab at this point and then select the Conformance Tests control again.
5. Run the first Initiating Imaging Gateway conformance test. That test should complete successfully and the test line turn green.
6. Assuming the step above worked, activate the button to run all of the tests. Each test will run in succession and the status should be green for each one.

Responding Imaging Gateway

1. Create a Test Session (eta).
   • Do not create any simulators by hand.
2. Select the Conformance Tests control and then Responding Imaging Gateway from the pane that is displayed.
3. Inside the Responding Imaging Gateway pane, activate the button to initialize the test environment.
4. Open the Test Context box. Make sure the proper Test Session (eta) is selected.
   • When the testing environment is created, an RIG simulator is automatically created. The name is $session__simulator-rig.
   • For the System Under test, select the RIG simulator that was automatically created (e.g., eta__simulator-rig).
   • I have had some issues with the tools just after this step. I recommend you close the ConfTests tab at this point and then select the Conformance Tests control again.
5. Run the first Responding Imaging Gateway conformance test. That test should complete successfully and the test line turn green.
6. Assuming that test runs successfully, activate the button to run all of the tests. Each test will run in succession, and the results should show green bars for each test.