9 Test failures investigation

9 Test failures investigation

The Sailfish Regression Test library usually has the following hierarchy:

· Test Library consisting of

·· Test Scripts consisting of

···Test Cases consisting of

····Test Actions.

If any Test Action fails, its Test Case and Test Script fail too.

On the Test Script page, the report on the Sailfish ‘Test scripts’ tab is highlighted with:

• Red if the script failed due to INIT FAILED / RUN FAILED;
• Yellow if the script was EXECUTED but has some failed cases;
• Green if the script was EXECUTED, and all the test cases passed or conditionally passed.

The report can be downloaded as a .zip archive or opened on a new web page via the Report Link.

On the report page, there is a list of all test cases that are a part of the script, with their descriptions. They are highlighted with:

• Red for failed test cases;
• Blue for passed test cases;
• Yellow for conditionally passed test cases.

Left click on a row with a Test Case name. A detailed report will open with information as follows:

• Start Time;
• Finish Time;
• Unique Case ID (if filled in matrix);
• Hash of Unique Case ID;
• View menu which provides an ability to choose if all cases/fields should be displayed or only the fields;
• The list of Actions in the Test Case with their descriptions.

Left click on a row with a Test Action name. A detailed report will open with information as follows:

1. A table with Action details (list of Action Parameters);
2. Action Status with the failure reason;
3. A table with Similar messages (for Receive or Retrieve action);
4. Tables with parameters of Similar messages with Expected and Actual values for each parameter (Highlighted with red if there is a discrepancy).

For the failed receive action the Comparison table with Expected and Actual values will be displayed as below.