Skip to content

Commit

Permalink
documentation(coverage): provide coverage example
Browse files Browse the repository at this point in the history
  • Loading branch information
ArnaudBuchholz committed Sep 28, 2023
1 parent b088bcd commit 869b2bb
Show file tree
Hide file tree
Showing 3 changed files with 89 additions and 16 deletions.
93 changes: 83 additions & 10 deletions docs/coverage.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,23 +2,96 @@

## Overview

_(Explain instrumentation)_
_(Explain how coverage data is collected and extracted)_
_(Explain report generation)_
Tests ensure that the code **behaves** as expected.
The code coverage checks which lines of code (function, branches...) are **executed** while running the tests.

## Legacy mode
If the code coverage shows that only 10% of the code is executed during your tests it means that 90% of the code is either not tested or useless.

**NOTE** : having 100% code coverage does not mean that the code is fully tested !

For instance, considering the function to test :

```javascript
function div (a, b) { return a / b; }
```

This assertion generates 100% code coverage :

```javascript
assert.notStrictEqual(div(4,2), 0);
```

But, in reality, it says almost nothing on the tested code.

## How does it work ?

The following is a summary of the article [REserve — Testing UI5 — Measuring code coverage](https://medium.com/@arnaud-buchholz/reserve-testing-ui5-measuring-code-coverage-ef303af051ef).

Three steps are required to measure code coverage :
1. **Instrumentation** of the code
2. **Execution** of the tests
3. **Extraction** and **consolidation** of code coverage

The first step is realized with [`nyc`](https://www.npmjs.com/package/nyc) which is a wrapper for [`istanbul`](https://www.npmjs.com/package/istanbul).
It rewrites the code in such a way that :

* The code behavior does **not** change
* While the code is executed, it keeps track of which functions, lines and conditions are evaluated.

`ui5-test-runner` takes care of the second step. Based on the mode and options, it will either substitute the source files with the instrumented ones or expect the source to provide coverage information *(see below)*.

Performance issues with UI5
While running, the runner extracts the code coverage information for each page (available within `window.top.__coverage__`) and stores it in the coverage temporary folder.

Introduce the hybrid mode where the runner serves the application files and @ui5/cli is used for `ressources/`
When all the test pages are executed, the coverage report is generated using two commands :

* `nyc merge` to merge the different coverage reports in a single one,
* `nyc report` to generate the report.

## Legacy mode

In this mode, source files are directly manipulated by `ui5-test-runner`.

## Remote mode

It is possible to extract code coverage using the remote mode.

### @ui5/middleware-code-coverage

Good *but* report generation does not work as expected
the runner will fetch files to build the report
The [`@ui5/middleware-code-coverage`](https://www.npmjs.com/package/@ui5/middleware-code-coverage) middleware is capable of instrumenting the source files on the fly (by adding `?instrument=true` to the URL of the file).

According to the documentation, the coverage information can be leveraged only from UI5 1.113 thanks to `qunit-coverage-istanbul.js` which adds the URL parameter dynamically.

`ui5-test-runner` reproduces the behavior by altering the URLs without `qunit-coverage-istanbul.js` making the coverage available for *any* version.

Here is an example of `ui5.yaml` configuration file to include the middleware.

```yaml
specVersion: '3.0'
metadata:
name: training-ui5con18-opa
type: application
server:
customMiddleware:
- name: "@ui5/middleware-code-coverage"
afterMiddleware: compression
configuration:
instrument:
produceSourceMap: true
coverageGlobalScope: "window.top"
coverageGlobalScopeFunc: false
excludePatterns:
- "test/"
- "resources/"
```
**Implementation note** : unlike the expected usage of the middleware, `ui5-test-runner` generates the coverage report. It forces the runner to download all the covered source files locally to ensure the report can be generated.

### coverage-proxy *(experimental)*

If the remote server does not provide instrumented source files, an experimental approach consists in using `ui5-test-runner` as a 'proxy' to get the files. It will instrument the sources on the fly.

### Future ?
Example :
```bash
ui5-test-runner --url https://ui5.sap.com/test-resources/sap/m/demokit/orderbrowser/webapp/test/testsuite.qunit.html --coverage -coverage-proxy --coverage-proxy-include webapp/* --coverage-proxy-exclude webapp/test --disable-ui5
```

ui5-test-runner runs as a proxy
2 changes: 1 addition & 1 deletion docs/usage.md
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,7 @@ Check additional information below.
-ccs, --coverage-check-statements <percent> [💻🔗] What % of statements must be covered (default: 0)
-s, --serve-only [flag] [💻🔗] Serve only (default: false)
--ui5 <url> [💻] UI5 url (default: "https://ui5.sap.com")
--no-ui5 [💻] Disable UI5 mapping (also disable libs)
--disable-ui5 [💻] Disable UI5 mapping (also disable libs) (default: false)
--libs <lib...> [💻] Library mapping (<relative>=<path> or <path>)
--mappings <mapping...> [💻] Custom mapping (<match>=<file|url>(<config>))
--cache <path> [💻] Cache UI5 resources locally in the given folder (empty to disable)
Expand Down
10 changes: 5 additions & 5 deletions mkdocs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -7,14 +7,14 @@ docs_dir: "docs" # default
site_dir: "site" # default
nav:
- Home: README.md
- Browser instantiation command: browser.md
- Configuration file: configuration.md
- Serving and testing the application (a.k.a. legacy mode): legacy.md
- Command line usage: usage.md
- Mapping v1 settings to v2: mapping_v1_v2.md
- Serving and testing the application (a.k.a. legacy mode): legacy.md
- Testing a "remote" application: testing.md
- Coverage extraction: coverage.md
- Configuration file: configuration.md
- Tips & tricks: tipsNtricks.md
- Command line usage: usage.md
- ui5-test-runner v2: v2.md
- Browser instantiation command: browser.md
- Automation Libraries:
- puppeteer: puppeteer.md
- jsdom: jsdom.md
Expand Down

0 comments on commit 869b2bb

Please sign in to comment.