Merge pull request #68 from Peter-Darton-i2/pr/document-circleci

Document CircleCI recommendations
spekt · Aug 8, 2023 · 330af00 · 330af00
2 parents 34467f2 + f67529c
commit 330af00
Show file tree

Hide file tree

Showing 7 changed files with 157 additions and 16 deletions.
diff --git a/README.md b/README.md
@@ -26,13 +26,16 @@ To use the logger, follow these steps:
 
 1. Add a reference to the [JUnit Logger](https://www.nuget.org/packages/JUnitXml.TestLogger) nuget package in test project
 2. Use the following command line in tests
-```
-> dotnet test --logger:junit
-```
+
+   ```none
+   > dotnet test --logger:junit
+   ```
+
 3. Test results are generated in the `TestResults` directory relative to the `test.csproj`
 
 A path for the report file can be specified as follows:
-```
+
+```none
 > dotnet test --logger:"junit;LogFilePath=test-result.xml"
 ```
 
@@ -43,7 +46,7 @@ A path for the report file can be specified as follows:
 All common options to the logger are documented [in the wiki][config-wiki]. E.g.
 token expansion for `{assembly}` or `{framework}` in result file. If you are writing multiple
 files to the same directory or testing multiple frameworks, these options can prevent
-test logs from over-writing eachother.
+test logs from over-writing each other.
 
 [config-wiki]: https://github.com/spekt/testlogger/wiki/Logger-Configuration
 
@@ -54,19 +57,19 @@ provide additional control over the xml file so that the logged test results can
 
 Platform Specific Recommendations:
 
-- [GitLab CI/CD Recomendation](/docs/gitlab-recommendation.md)
-- [Jenkins Recomendation](/docs/jenkins-recommendation.md) 
-- [CircleCI Recomendation](/docs/circleci-recommendation.md)
+- [GitLab CI/CD Recommendation](/docs/gitlab-recommendation.md)
+- [Jenkins Recommendation](/docs/jenkins-recommendation.md)
+- [CircleCI Recommendation](/docs/circleci-recommendation.md)
 
 After the logger name, command line arguments are provided as key/value pairs with the following general format. **Note** the quotes are required and key names are case sensitive.
 
-```
+```none
 > dotnet test --test-adapter-path:. --logger:"junit;key1=value1;key2=value2"
 ```
 
 #### MethodFormat
 
-This option alters the `testcase name` attribute. By default, this contains only the method. Class, will add the class to the name. Full, will add the assembly/namespace/class to the method. 
+This option alters the `testcase name` attribute. By default, this contains only the method. Class, will add the class to the name. Full, will add the assembly/namespace/class to the method.
 
 We recommend this option for [GitLab](/docs/gitlab-recommendation.md) users.
 
@@ -78,9 +81,9 @@ We recommend this option for [GitLab](/docs/gitlab-recommendation.md) users.
 
 #### FailureBodyFormat
 
-When set to default, the body of a `failure` element will contain only the exception which is captured by vstest. Verbose will prepend the body with 'Expected X, Actual Y' similar to how it is displayed in the standard test output. 'Expected X, Actual Y' are normally only contained in the failure message. Additionally, Verbose will include standard output from the test in the failure message. 
+When set to default, the body of a `failure` element will contain only the exception which is captured by vstest. Verbose will prepend the body with 'Expected X, Actual Y' similar to how it is displayed in the standard test output. 'Expected X, Actual Y' are normally only contained in the failure message. Additionally, Verbose will include standard output from the test in the failure message.
 
-We recommend this option for [GitLab](/docs/gitlab-recommendation.md) users.
+We recommend this option for [GitLab](/docs/gitlab-recommendation.md) and [CircleCI](/docs/circleci-recommendation.md) users.
 
 ##### Allowed Values
 

diff --git a/docs/assets/circleci-test-collapsed-with-methodname-class.png b/docs/assets/circleci-test-collapsed-with-methodname-class.png
diff --git a/docs/assets/circleci-test-collapsed-with-methodname-default.png b/docs/assets/circleci-test-collapsed-with-methodname-default.png
diff --git a/docs/assets/circleci-test-expanded-with-failure-default.png b/docs/assets/circleci-test-expanded-with-failure-default.png
diff --git a/docs/assets/circleci-test-expanded-with-failure-verbose.png b/docs/assets/circleci-test-expanded-with-failure-verbose.png
diff --git a/docs/circleci-recommendation.md b/docs/circleci-recommendation.md
@@ -1,3 +1,141 @@
 # CircleCI Recommendation
 
-**TODO** If you are a Circle CI user who is willing to help, please see Issue #37
+## Summary
+
+```yml
+- run:
+   name: Run Tests
+   command: |
+     dotnet test ... --logger:"junit;LogFilePath=TestResults/test-result.xml;FailureBodyFormat=Verbose;MethodFormat=Class"
+- store_test_results:
+   path: TestResults/
+- store_artifacts:
+   path: TestResults/
+```
+
+See also: [this sample repository][circleci-windows-project-example].
+
+## Details
+
+CircleCI uses just a few pieces of the JUnit XML report to generate the displayed user interface.
+A basic example `junit.xml` file which can be parsed by CircleCI can be seen
+[here][circleci-junit-xml-report].
+
+For each failed test (i.e. for each JUnit `<testcase>` element containing a failure or an error),
+CircleCI only shows the testcase's failure/error message,
+any text within the failure/error element,
+plus any `<system-out>` or `<system-err>` text.
+It does not show the testcase's properties.
+It does not show data from the testsuite.
+
+By default, the logger records test console output in the JUnit testsuite,
+which CircleCI ignores;
+this can be improved upon by setting `FailureBodyFormat` when invoking the logger.
+Setting MethodFormat provides no additional information or functionality.
+
+### FailureBodyFormat
+
+Setting `FailureBodyFormat=Verbose` when invoking the logger ensures that
+any test console output is visible in the CircleCI UI for (failed) tests.
+This can provide important/useful information,
+especially if the test failure message is insufficiently informative on its own.
+
+- `Default`:
+
+  ![FailureBodyFormat Default](assets/circleci-test-expanded-with-failure-default.png)
+
+- `Verbose`:
+
+  ![FailureBodyFormat Verbose](assets/circleci-test-expanded-with-failure-verbose.png)
+
+### MethodFormat
+
+CircleCI shows the class names of each failed test so setting
+`MethodFormat=Class` or `MethodFormat=Full`
+provides no further information on the CircleCI UI;
+the effect is purely cosmetic and is a matter of personal preference.
+
+- `Default`:
+
+  ![MethodFormat Default](assets/circleci-test-collapsed-with-methodname-default.png)
+
+- `Class` and `Full`:
+
+  ![MethodFormat Class](assets/circleci-test-collapsed-with-methodname-class.png)
+
+### Store Test Results
+
+To enable CircleCI to show test results on the job's page,
+to interpret generated test result files to be used for gathering
+[Test Insights][circleci-test-insights]
+and
+[Test Splitting][circleci-test-splitting],
+the files must be stored using the
+[`store_test_results` step][circleci-store-test-results-step].
+
+```yml
+- store_test_results:
+   path: TestResults/
+```
+
+### Store Artifacts
+
+To enable further troubleshooting,
+the test results file should _also_ be stored as an artifact using the
+[`store_artifacts` step][circleci-store-artifacts-step].
+This allows the user to view and download the contents of the raw `*.xml` file,
+which is useful where the user requires information from the file that CircleCI does not display.
+
+```yml
+- store_artifacts:
+   path: TestResults/
+```
+
+### Test Splitting
+
+CircleCI can enable a suite a tests to be split across multiple executors in parallel by using (timing) data logged by this junit logger.
+
+The simplified summary (above) does not include this functionality but this is recommended for non-trivial test workloads,
+especially where the test execution time significantly exceeds the time taken to set up ready for the first test.
+This technique reduces the total (elapsed-time) duration of the tests and thus provides a faster test feedback loop.
+
+For an example of a CircleCI Windows project that demonstrates parallel test execution, take a look at
+[this sample repository][circleci-windows-project-example].
+
+-----
+
+## Footnote
+
+As of July 20 2023:
+
+- Only the data in a `<testcase>` element is interpreted by CircleCI.
+  - e.g. `<system-out>` from within a `<testcase>` is shown.
+- Data outside of a `<testcase>` element is not interpreted by CircleCI.
+- The `<properties>` from a `<testcase>` aren't interpreted by CircleCI.
+- By default, the logger only puts console text (`<system-out>` and `<system-err>`) in elements at the `<testsuite>` level (not into `<testcase>`) so CircleCI  does not interpret it.
+
+e.g.
+
+```xml
+<testsuites>
+  <testsuite>
+    <testcase name="Displayed in bold on the CircleCI WebUI" classname="Also shown" time="Used for test splitting">
+      <properties>
+        <property name="CircleCI" value="Does not show these" />
+      </properties>
+      <failure message="This can be seen in CircleCI">So can this</failure>
+      <system-out>This is also shown by CircleCI<system-out>
+      <system-err>Same for system-err too</system-err>
+    </testcase>
+    <system-out>CircleCI ignores data here</system-out>
+    <system-err>Same for system-err too</system-err>
+  </testsuite>
+</testsuites>
+```
+
+[circleci-junit-xml-report]: https://circleci.com/docs/use-the-circleci-cli-to-split-tests/#junit-xml-reports
+[circleci-test-insights]: https://circleci.com/docs/insights-tests/
+[circleci-test-splitting]: https://circleci.com/docs/use-the-circleci-cli-to-split-tests/#split-by-timing-data
+[circleci-store-test-results-step]: https://circleci.com/docs/configuration-reference/#storetestresults
+[circleci-store-artifacts-step]: https://circleci.com/docs/configuration-reference/#storeartifacts
+[circleci-windows-project-example]: https://github.com/jenny-miggin/circleci-demo-windows-test-splitting
diff --git a/docs/gitlab-recommendation.md b/docs/gitlab-recommendation.md
@@ -4,7 +4,7 @@ GitLab uses just a few pieces of the XML report to generate the displayed user i
 
 ## Default Display
 
-The summary view shows passing and failing tests by method name (including any parameters). GitLab generates its summary of New failing tests, existing failing tests, and newly passing tests based on this string. 
+The summary view shows passing and failing tests by method name (including any parameters). GitLab generates its summary of New failing tests, existing failing tests, and newly passing tests based on this string.
 
 ![Default Test Summary](assets/gitlab-test-summary-with-default-option.png)
 
@@ -14,7 +14,7 @@ The popup displayed when clicking on a test, shows only the body of the failure.
 
 ## Improved Output With Formatting Options
 
-The test summary can be modified by setting the method format. The option below used 'MethodFormat=Class'. This can be particularly helpful when using test fixture data (i.e. there are paramters passed to the class) which you would like to display.
+The test summary can be modified by setting the method format. The option below used 'MethodFormat=Class'. This can be particularly helpful when using test fixture data (i.e. there are parameters passed to the class) which you would like to display.
 
 ![Test Summary with Class Option](assets/gitlab-test-summary-with-class-option.png)
 
@@ -38,7 +38,7 @@ Test:
     reports:
       junit: 
        - .\artifacts\*test-result.xml
-```       
+```
 
 ## Notes