Skip to content

Commit 7ddbcdc

Browse files
committed
Think before you force :(
1 parent b7d295b commit 7ddbcdc

File tree

1 file changed

+30
-32
lines changed

1 file changed

+30
-32
lines changed

docs/tutorial/writing-tests/test-plan.rst

Lines changed: 30 additions & 32 deletions
Original file line numberDiff line numberDiff line change
@@ -1,18 +1,18 @@
11
.. _test_plan:
22

3-
=================
4-
Writing Test Plan
5-
=================
3+
===================
4+
Writing A Test Plan
5+
===================
66

77
This tutorial will guide you in writing a test plan to test the Network
8-
connection in your machine. We will do this by re-using tests that are already
8+
connection on your machine. We will do this by re-using tests that are already
99
available in the ``tutorial provider`` and that you got to write yourself in
1010
the previous tutorial.
1111

1212
Inclusions
1313
==========
1414

15-
When we want a test plan to contain a test what we do in Checkbox is including
15+
When we want a test plan to contain a test, what we do in Checkbox is including
1616
it. There are a few kinds of inclusions but they all have the same underlying
1717
purpose: to tell Checkbox that we want to run something. Start by creating a
1818
new test plan in the same provider we created in the previous tutorial.
@@ -24,7 +24,7 @@ new test plan in the same provider we created in the previous tutorial.
2424
``providers/tutorial/units/test-plan.pxu``
2525

2626
We now have a convenient container where to put all tests we previously
27-
developed, let's include them in a new ``test plan``.
27+
developed, let's include them in a new ``test plan``:
2828

2929
.. code-block::
3030
@@ -48,7 +48,7 @@ tests:
4848
☑ : Test that the network speed is acceptable
4949
5050
Note how, as we previously saw, Checkbox automatically pulled the resource
51-
job need. This operation, as we previously mention, is not the safe way to go
51+
job needed. This operation, as we previously mentioned, is not the safe way to go
5252
about dependency management, it is just an aid Checkbox gives you while
5353
developing. Jobs that are automatically pulled are placed in a random spot in
5454
the list where you may already have broken the thing you are trying to fetch
@@ -110,11 +110,11 @@ Status Overrides
110110
The certification status of a job can be defined in its definition. This is
111111
useful, but limiting, as one may want the same test to be a certification
112112
blocker in one test plan while not in another. Checkbox supports overrides in
113-
test plan that allow you to change the certification status (common) or the
113+
test plans that allow you to change the certification status (common) or the
114114
category (uncommon) of a job in that specific test plan.
115115

116116
Going back to the test plan we just defined let's add the following and see the
117-
effect in the expand output:
117+
effect in the ``expand`` output:
118118

119119
.. code-block::
120120
@@ -129,7 +129,7 @@ effect in the expand output:
129129
apply blocker to network_available
130130
131131
132-
Running expand we can see that the certification status changed:
132+
Running ``expand`` we can see that the certification status changed:
133133

134134
.. code-block::
135135
@@ -161,7 +161,7 @@ Bootstrap Inclusions
161161
====================
162162

163163
As we have previously discussed, resources are the backbone of Checkbox
164-
information gathering. Using the data they generate jobs are skipped or ran and
164+
information gathering. Using the data they generate, jobs are skipped or ran and
165165
templates are instantiated. Although Checkbox does try to pull all resources
166166
and dependencies you may need into a test plan automatically, jobs may
167167
interfere or break resources so, ideally, we would like to run them before
@@ -180,7 +180,7 @@ in the ``bootstrap_include`` section:
180180
unit: test plan
181181
id: tutorial-extended
182182
_name: Extended Tutorial Test Plan
183-
bootstrap-include:
183+
bootstrap_include:
184184
network_iface_info
185185
include:
186186
network_available
@@ -203,7 +203,7 @@ Let's update the test plan including it:
203203
unit: test plan
204204
id: tutorial-extended
205205
_name: Extended Tutorial Test Plan
206-
bootstrap-include:
206+
bootstrap_include:
207207
network_iface_info
208208
include:
209209
network_available_interface
@@ -212,20 +212,18 @@ Let's update the test plan including it:
212212
certification_status_overrides:
213213
apply blocker to network_available
214214
215-
When we run expand on the test plan, two important changes occur in the output:
216-
217-
First, the resource job is no longer visible - this is expected! The bootstrap
218-
section of a test plan is meant to gather essential data before the main test
219-
execution but is not composed of actual tests, so the jobs there are excluded
220-
from the expand command.
221-
222-
Second, our newly added template wasn't expanded. This happens because a
223-
template is expanded on the result of a resource, and only running the resource
224-
can give that output (that is often specific to one machine!). If we want to
225-
see what would the test plan expand to on the current machine, we can use
226-
``list-bootstrapped``:
215+
When we run ``expand`` on the test plan, two important changes occur in the
216+
output:
227217

228-
Try to run the following
218+
- First, the resource job is no longer visible – this is expected! The
219+
bootstrap section of a test plan is meant to gather essential data before the
220+
main test execution but is not composed of actual tests, so the jobs there
221+
are excluded from the expand command.
222+
- Second, our newly added template wasn't expanded. This happens because a
223+
template is expanded on the result of a resource, and only running the
224+
resource can give that output (that is often specific to one machine!). If we
225+
want to see all the jobs that would be executed on the current machine if we
226+
ran that test plan, we can use ``list-bootstrapped``:
229227

230228
.. code-block::
231229
@@ -333,7 +331,7 @@ missing:
333331
Excluding a test via ``exclude`` in the test plan is different from using
334332
``exclude`` in the launcher. If you use ``exclude`` in the launcher, you
335333
are modifying the test plan, so it will not be accepted as a submission on
336-
C3, while if you use ``exclude`` in a test plan, you are creating a new,
334+
C3, whereas if you use ``exclude`` in a test plan, you are creating a new,
337335
different test plan.
338336

339337
Using exclude to remove tests is one mechanism to customize your test plan, but
@@ -343,10 +341,10 @@ about why you are excluding those tests, maybe some need an updated definition!
343341

344342
.. warning::
345343
While ``exclude`` is a list of regexes, so you can use a regex to exclude
346-
jobs, you should most likely avoid doing that as you may inadvertently lose
347-
more jobs (and time) than you were aiming for. Try to always precisely match
344+
jobs, you should most likely avoid doing that as you may inadvertently deselect
345+
more jobs than you were aiming for. Try to always precisely match
348346
what you want to exclude, for templates, for example, use the template id
349-
whenever you can instead of regex matching the generated id
347+
whenever you can instead of regex matching the generated id.
350348

351349
Mandatory Inclusions
352350
====================
@@ -380,5 +378,5 @@ See how the output of ``list-bootstrapped`` is unaffected.
380378
com.canonical.certification::info/systemd-analyze-critical-chain
381379
[...]
382380
383-
The reason is that all tests in the ``submission-cert-automated`` are mandatory
384-
includes
381+
The reason is that all tests in the ``submission-cert-automated`` nested part are mandatory
382+
includes: they will be executed regardless of any other rule in your test plan.

0 commit comments

Comments
 (0)