|
| 1 | +## Percy visual testing |
| 2 | + |
| 3 | +We use [Percy](https://percy.io) for visual testing. Percy tests are run against pull requests to |
| 4 | +ensure that PRs do not introduce visual regressions. Your PR will be tested by Percy if it meets the following conditions: |
| 5 | + |
| 6 | +- PR is against the `main` branch |
| 7 | +- One of the following is true: |
| 8 | + - PR passes Percy selectivity filters |
| 9 | + - PR makes changes to any of the following directories or files: |
| 10 | + - `scss/` |
| 11 | + - `templates/docs/examples/` |
| 12 | + - `templates/_macros/` |
| 13 | + - `tokens/` |
| 14 | + - `sd.config.json` |
| 15 | + - PR is not a draft |
| 16 | + - PR is labeled with "Review: Percy needed" |
| 17 | + - Note that a Percy (labeled) workflow will be triggered on each labelling and synchronisation event on PRs to `main`. They will be skipped shortly after starting if the PR does not have the "Review: Percy needed" label. |
| 18 | + |
| 19 | +To ensure optimal Percy usage, we suggest the following PR flow: |
| 20 | + |
| 21 | +1. Open the PR (against `main`) as a draft. Ensure that all non-Percy PR status checks pass. |
| 22 | +2. Apply the "Review: Percy needed" label to the PR to trigger a Percy test. |
| 23 | +3. Review the initial Percy build. After a few minutes, it will be available on our [Percy dashboard](https://percy.io/bb49709b/vanilla-framework). You can also click the Percy status check on the PR to navigate directly to the test results. |
| 24 | + |
| 25 | +<img src="https://assets.ubuntu.com/v1/a4b79af3-percy_status_check.png" alt="Percy status check" width="400"> |
| 26 | + |
| 27 | +4. If there are additional changes needed to the PR through the review process, you can remove the "Review: Percy needed" |
| 28 | + label and mark the PR as a draft to prevent additional Percy tests from running. Any commits you add to the PR will not trigger a Percy test, as long as the PR is a draft and does not have the "Review: Percy needed" label. |
| 29 | +5. Once the PR is ready for final review, remove the draft status and reapply the "Review: Percy needed" label to trigger |
| 30 | + a final Percy test. |
| 31 | +6. If the Percy test passes, apply "Review: Percy +1" to indicate that the PR has passed Percy testing. |
| 32 | + |
| 33 | +### Workflows |
| 34 | + |
| 35 | +#### Baseline Workflow |
| 36 | + |
| 37 | +The [baseline workflow](../.github/workflows/percy-baseline.yml) runs a Percy test on every push to `main`, and sets the results |
| 38 | +as the new Percy baseline that all future PRs will be compared against. |
| 39 | + |
| 40 | +#### Prepare Workflow |
| 41 | + |
| 42 | +The [prepare workflow](../.github/workflows/percy-prepare.yml) checks out a merge commit of a PR and `main`, then |
| 43 | +uploads all files in the PR that have any affects on the visual appearance of the design system as a GitHub artefact. |
| 44 | + |
| 45 | +#### PR Snapshot Workflow |
| 46 | + |
| 47 | +The [snapshot workflow](../.github/workflows/pr-percy-snapshots.yml) downloads the artefacts from the [prepare workflow](#prepare-workflow) |
| 48 | +and calls the [Percy snapshot action](../.github/actions/percy-snapshot/action.yml) to build, run, & visually test Vanilla. |
| 49 | + |
| 50 | +The PR snapshot workflow is always called after the completion of the [prepare workflow](#prepare-workflow). |
| 51 | + |
| 52 | +### Example snapshots |
| 53 | + |
| 54 | +Percy is run by executing `yarn percy` or `npm run percy` in the root of the project. The [Percy CLI](https://github.com/percy/cli) |
| 55 | +runs [snapshots.js](../snapshots.js) which returns a list of all the snapshots that Percy will take. |
| 56 | + |
| 57 | +Percy uses our [examples](https://vanillaframework.io/docs/examples) to take snapshots of all components, patterns, and utilities in Vanilla. |
| 58 | + |
| 59 | +[snapshots.js](../snapshots.js) will take a snapshot of an example template file if the following conditions are met: |
| 60 | + |
| 61 | +- Is an HTML file in `templates/docs/examples/` |
| 62 | +- File is not a partial (does not start with `_`) |
| 63 | +- File is not included in a combined example (is not in the same directory as a combined example or in a directory that descends from a directory with a combined example) |
| 64 | + |
| 65 | +#### Snapshot widths |
| 66 | + |
| 67 | +Each snapshot object returned by [snapshots.js](../snapshots.js) has a `widths` array property that specifies the widths (in pixels) at which the snapshot should be captured. |
| 68 | +The following table shows the widths at which snapshots are captured and which examples they apply to: |
| 69 | + |
| 70 | +| Device | Width (px) | Captured on | |
| 71 | +| ------- | ---------- | ---------------------------------------------------------------------------------------------------------------------------------------------- | |
| 72 | +| Desktop | 1,280 | All examples | |
| 73 | +| Tablet | 800 | All examples with `responsive` in their names, and [combined examples](#combined-examples) that include a `responsive` file in their directory | |
| 74 | +| Mobile | 375 | Default theme only (currently `light`) | |
| 75 | + |
| 76 | +Each width is captured as a separate screenshot, and counts as an additional screenshot in Percy's pricing model. |
| 77 | + |
| 78 | +#### Snapshot browsers |
| 79 | + |
| 80 | +Percy can be configured to take snapshots in multiple browsers. Each additional browser counts as an additional set of screenshots in Percy's pricing model. |
| 81 | +Vanilla is currently only tested on Chrome. To add additional browsers, view the [Percy project settings page](https://percy.io/bb49709b/vanilla-framework/settings). |
| 82 | + |
| 83 | +#### Combined examples |
| 84 | + |
| 85 | +Combined examples are a way to show all the variants of a component or pattern on the same page. This helps us keep our |
| 86 | +Percy usage down by only taking one snapshot of a component or pattern, rather than one for each variant. |
| 87 | +Combined examples follow a naming convention: they must be named `combined.html`. |
| 88 | +If `combined.html` is found in a directory, [snapshots.js](../snapshots.js) will assume that every example in that directory (and its subdirectories) is rendered in the `combined.html` file. |
| 89 | + |
| 90 | +**Combined examples do not currently include the JavaScript or CSS that is included in individual examples by `{% block script %}` and `{% block style %}` tags.** |
| 91 | + |
| 92 | +##### Creating a combined example |
| 93 | + |
| 94 | +1. Create a Jinja template file named `combined.html` in any subdirectory of `templates/docs/examples/` |
| 95 | +2. In your `content` block, create a `with` block that sets the flag `is_combined` to `true`. This will ensure that the [base example template](../templates/_layouts/examples.html) renders the example without the `{% block script %}` and `{% block style %}` tags and adds links to the individual examples. |
| 96 | +3. Include the examples you want to combine in the `combined.html` file using `{% include 'path/to/example.html' %}` inside the `with` block. It is good practice to separate the examples in `<section>` tags. |
| 97 | + |
| 98 | +For example, see the [combined example for the button pattern](../templates/docs/examples/patterns/buttons/combined.html). |
| 99 | + |
| 100 | +##### Spacing below examples |
| 101 | + |
| 102 | +If an example's content is expected to overflow its body (such as a dropdown or tooltip), you can use the `spacing_below` parameter to add space below the example. |
| 103 | +`spacing_below` `rems` will be added beneath the example as `margin-bottom`. |
| 104 | + |
| 105 | +For example: |
| 106 | + |
| 107 | +```html |
| 108 | +<!-- Add `12rem` of space below the example --> |
| 109 | +{% with spacing_below = 12 %} |
| 110 | +<section>{% include 'docs/examples/patterns/contextual-menu/default.html' %}</section> |
| 111 | +{% endwith %} |
| 112 | +``` |
| 113 | + |
| 114 | +### Testing |
| 115 | + |
| 116 | +[snapshots.test.js](../tests/snapshots.test.js) is a [Jest](https://jestjs.io/) testing file used to test the output of the [snapshots.js](../snapshots.js) file. |
| 117 | +It tests that the snapshots are correctly generated. It is run as part of our wider testing script (`yarn test`), but you can also execute it directly by running `npx jest tests/snapshots.test.js`. |
| 118 | + |
| 119 | +[snapshots.js](../snapshots.js) considers `combined.html` to require responsive snapshots if a sibling or descendant file contains `responsive` in its name. |
| 120 | +[snapshots.test.js](../tests/snapshots.test.js) does not read the filesystem to decide whether a combined snapshot is responsive. |
| 121 | +If you add a combined example that must be captured responsively, you must add the path of that example to `RESPONSIVE_COMBINED_EXAMPLES` in [snapshots.test.js](../tests/snapshots.test.js). |
| 122 | + |
| 123 | +### PR File Inclusion |
| 124 | + |
| 125 | +Only some of a pull request's files are represented in each pull request snapshot test. Depending on which files need to be changed, |
| 126 | +you may need to make a separate pull request to make changes to `main` before a pull request will use your changes in Percy testing. |
| 127 | + |
| 128 | +#### Files included in Percy tests |
| 129 | + |
| 130 | +- Visual files: |
| 131 | + - `scss/` |
| 132 | + - `templates/docs/examples/` |
| 133 | + - `templates/_macros/` |
| 134 | + - `tokens/` |
| 135 | + - `sd.config.json` |
| 136 | +- Workflows that are run on the `pull_request` trigger, including (but not limited to): |
| 137 | + - [Prepare workflow](../.github/workflows/percy-prepare.yml) |
| 138 | + - [Prepare (labelled) workflow](../.github/workflows/pr-percy-prepare-label.yml) |
| 139 | + - [Prepare (pushed) workflow](../.github/workflows/pr-percy-prepare-push.yml) |
| 140 | + |
| 141 | +Changes to other files should be added to a separate PR and merged before testing a PR that requires their changes. |
| 142 | +Otherwise, the changes will not be included in Percy tests. |
| 143 | +This notably includes: |
| 144 | + |
| 145 | +- Dependencies |
| 146 | + - `Dockerfile` |
| 147 | + - `package.json` |
| 148 | + - `requirements.txt` |
| 149 | +- Workflows that are run on the `workflow_run` trigger: |
| 150 | + - [PR snapshots workflow](../.github/workflows/pr-percy-snapshots.yml) |
| 151 | + - [Snapshot action](../.github/actions/percy-snapshot/action.yml) |
| 152 | +- `snapshots.js` and `snapshots.test.js` |
| 153 | + |
| 154 | +### Additional resources |
| 155 | + |
| 156 | +- [Percy CLI documentation](https://www.browserstack.com/docs/percy/take-percy-snapshots/snapshots-via-cli#advanced-options) |
| 157 | +- [Percy environment variables](https://www.browserstack.com/docs/percy/get-started/set-env-var#Percy_SDK) |
| 158 | +- [Percy workflow blog post](https://ubuntu.com/blog/visual-testing-github-actions-migration-test-optimisation) |
0 commit comments