1
1
.. _test_plan :
2
2
3
- =================
4
- Writing Test Plan
5
- =================
3
+ ===================
4
+ Writing A Test Plan
5
+ ===================
6
6
7
7
This tutorial will guide you in writing a test plan to test the Network
8
- connection in your machine. We will do this by re-using tests that are already
8
+ connection on your machine. We will do this by re-using tests that are already
9
9
available in the ``tutorial provider `` and that you got to write yourself in
10
10
the previous tutorial.
11
11
12
12
Inclusions
13
13
==========
14
14
15
- When we want a test plan to contain a test what we do in Checkbox is including
15
+ When we want a test plan to contain a test, what we do in Checkbox is including
16
16
it. There are a few kinds of inclusions but they all have the same underlying
17
17
purpose: to tell Checkbox that we want to run something. Start by creating a
18
18
new test plan in the same provider we created in the previous tutorial.
@@ -24,7 +24,7 @@ new test plan in the same provider we created in the previous tutorial.
24
24
``providers/tutorial/units/test-plan.pxu ``
25
25
26
26
We now have a convenient container where to put all tests we previously
27
- developed, let's include them in a new ``test plan ``.
27
+ developed, let's include them in a new ``test plan ``:
28
28
29
29
.. code-block ::
30
30
48
48
☑ : Test that the network speed is acceptable
49
49
50
50
Note how, as we previously saw, Checkbox automatically pulled the resource
51
- job need . This operation, as we previously mention , is not the safe way to go
51
+ job needed . This operation, as we previously mentioned , is not the safe way to go
52
52
about dependency management, it is just an aid Checkbox gives you while
53
53
developing. Jobs that are automatically pulled are placed in a random spot in
54
54
the list where you may already have broken the thing you are trying to fetch
@@ -110,11 +110,11 @@ Status Overrides
110
110
The certification status of a job can be defined in its definition. This is
111
111
useful, but limiting, as one may want the same test to be a certification
112
112
blocker in one test plan while not in another. Checkbox supports overrides in
113
- test plan that allow you to change the certification status (common) or the
113
+ test plans that allow you to change the certification status (common) or the
114
114
category (uncommon) of a job in that specific test plan.
115
115
116
116
Going back to the test plan we just defined let's add the following and see the
117
- effect in the expand output:
117
+ effect in the `` expand `` output:
118
118
119
119
.. code-block ::
120
120
@@ -129,7 +129,7 @@ effect in the expand output:
129
129
apply blocker to network_available
130
130
131
131
132
- Running expand we can see that the certification status changed:
132
+ Running `` expand `` we can see that the certification status changed:
133
133
134
134
.. code-block ::
135
135
@@ -161,7 +161,7 @@ Bootstrap Inclusions
161
161
====================
162
162
163
163
As we have previously discussed, resources are the backbone of Checkbox
164
- information gathering. Using the data they generate jobs are skipped or ran and
164
+ information gathering. Using the data they generate, jobs are skipped or ran and
165
165
templates are instantiated. Although Checkbox does try to pull all resources
166
166
and dependencies you may need into a test plan automatically, jobs may
167
167
interfere or break resources so, ideally, we would like to run them before
@@ -180,7 +180,7 @@ in the ``bootstrap_include`` section:
180
180
unit: test plan
181
181
id: tutorial-extended
182
182
_name: Extended Tutorial Test Plan
183
- bootstrap-include :
183
+ bootstrap_include :
184
184
network_iface_info
185
185
include:
186
186
network_available
@@ -203,7 +203,7 @@ Let's update the test plan including it:
203
203
unit: test plan
204
204
id: tutorial-extended
205
205
_name: Extended Tutorial Test Plan
206
- bootstrap-include :
206
+ bootstrap_include :
207
207
network_iface_info
208
208
include:
209
209
network_available_interface
@@ -212,20 +212,18 @@ Let's update the test plan including it:
212
212
certification_status_overrides:
213
213
apply blocker to network_available
214
214
215
- When we run expand on the test plan, two important changes occur in the output:
216
-
217
- First, the resource job is no longer visible - this is expected! The bootstrap
218
- section of a test plan is meant to gather essential data before the main test
219
- execution but is not composed of actual tests, so the jobs there are excluded
220
- from the expand command.
221
-
222
- Second, our newly added template wasn't expanded. This happens because a
223
- template is expanded on the result of a resource, and only running the resource
224
- can give that output (that is often specific to one machine!). If we want to
225
- see what would the test plan expand to on the current machine, we can use
226
- ``list-bootstrapped ``:
215
+ When we run ``expand `` on the test plan, two important changes occur in the
216
+ output:
227
217
228
- Try to run the following
218
+ - First, the resource job is no longer visible – this is expected! The
219
+ bootstrap section of a test plan is meant to gather essential data before the
220
+ main test execution but is not composed of actual tests, so the jobs there
221
+ are excluded from the expand command.
222
+ - Second, our newly added template wasn't expanded. This happens because a
223
+ template is expanded on the result of a resource, and only running the
224
+ resource can give that output (that is often specific to one machine!). If we
225
+ want to see all the jobs that would be executed on the current machine if we
226
+ ran that test plan, we can use ``list-bootstrapped ``:
229
227
230
228
.. code-block ::
231
229
@@ -333,7 +331,7 @@ missing:
333
331
Excluding a test via ``exclude `` in the test plan is different from using
334
332
``exclude `` in the launcher. If you use ``exclude `` in the launcher, you
335
333
are modifying the test plan, so it will not be accepted as a submission on
336
- C3, while if you use ``exclude `` in a test plan, you are creating a new,
334
+ C3, whereas if you use ``exclude `` in a test plan, you are creating a new,
337
335
different test plan.
338
336
339
337
Using exclude to remove tests is one mechanism to customize your test plan, but
@@ -343,10 +341,10 @@ about why you are excluding those tests, maybe some need an updated definition!
343
341
344
342
.. warning ::
345
343
While ``exclude `` is a list of regexes, so you can use a regex to exclude
346
- jobs, you should most likely avoid doing that as you may inadvertently lose
347
- more jobs (and time) than you were aiming for. Try to always precisely match
344
+ jobs, you should most likely avoid doing that as you may inadvertently deselect
345
+ more jobs than you were aiming for. Try to always precisely match
348
346
what you want to exclude, for templates, for example, use the template id
349
- whenever you can instead of regex matching the generated id
347
+ whenever you can instead of regex matching the generated id.
350
348
351
349
Mandatory Inclusions
352
350
====================
@@ -380,5 +378,5 @@ See how the output of ``list-bootstrapped`` is unaffected.
380
378
com.canonical.certification::info/systemd-analyze-critical-chain
381
379
[...]
382
380
383
- The reason is that all tests in the ``submission-cert-automated `` are mandatory
384
- includes
381
+ The reason is that all tests in the ``submission-cert-automated `` nested part are mandatory
382
+ includes: they will be executed regardless of any other rule in your test plan.
0 commit comments