A Behaviour Driven Development (BDD) methodology was introduced for testing the Store. The option for this methodology ensures meeting the goals of (i) validating the Store behaviour from an API consumer's perspective and (ii) serve as documentation for describing the features available and the intended operational scenarios.
- Python 3
- radish Behavior Driven Development tool for Python
- Requests for HTTP requests and responses
- dictdiffer to compare expected versus actual JSON responses
- NodeJS for the modules:
- canned to mock the vNSF Orchestrator API
- cucumber-html-reporter to beautify test reports
- argv for command line argument parsing
The environment requirements are defined in the docker/requirements-qa.txt file.
To run the Quality Assurance (QA) environment from a teardown starting point:
cd docker
./run.sh --qaOnce the testing finishes a report is generated and the final output presents the location where the report can be found. This signals the end of the testing operation and is as follows:
===
=== Tests report is at /usr/share/dev/store/test/reports
===To view a beautified version of the report please open the /usr/share/dev/store/test/reports/report.html file.
As for the BDD methodology features (or specifications) for what the Store should provide are defined along with scenarios ("tests") of applicability. All this is defined using Gherkin notation and radish as the BDD support tool.
The QA base folder is located at test.
Features are described in *.features files and can be found in the features folder.
Steps are reusable portions used in scenarios. Its behaviour is defined in the steps*.py Python files under the features/radish folder.
Any validation requires a controlled and known environment. For this input data and expected output is paramount to determine whether abnormal behaviour is present and pinpoint any detours.
The input data is stored in the resources/input-data folder and takes the form best suited for the operation in question, be it a data file or a compressed file.
The expected output is stored in the resources/expected-output folder and takes the form of a JSON file which mimics the Store response to an HTPP request done through its API. This file may be an exact match of the expected response or only the bits that won't change over time. This is defined as discarding output.
In order to identify any detour from the expected behaviour an exact match between actual and expected output must be carried out. Sometimes this purpose is defeated by the ever-changing nature of some response elements (like identifiers and dates for instance) which will take on a different value every time a scenario runs.
To cope with such requirement the JSON file with the expected response contents uses a special notation to state what is to be ignored and what must be matched. Anything else present in the response and not stated in this notation shall render the validation failed and thus the scenario unsuccessful.
An example of the JSON notation to use when wanting to discard some output from the Store API is presented next.
{
"ignore": [
"root['_id']",
"root['_etag']",
"root['_created']",
"root['_updated']",
"root['_links']"
],
"expected": {
"_status": "OK"
}
}The entire QA set is commanded by the test/run.sh script. This takes care of validating all the features and provide a beautified HTML report of the execution. This report lives at /usr/share/dev/store/test/reports/report.html.
When wanting to run a subset of features or scenarios one can open a shell to the QA container and run only the intended ones. This is accomplished by simply open a shell with the QA container and run the desired set. To open a shell do:
docker container exec -it docker_qa_1 bash
cd ${FOLDER_TESTS_FEATURES}From this point on one can select what to run and produce a beautified report of the run.
It is possible to run only a specific set of features or scenarios from the entire selection available. This is done through:
-
running a specific features file
radish --cucumber-json </path/to/report/output.json> health-checks.feature -
identify the group(s) to run by its tag(s)
radish --cucumber-json </path/to/report/output.json> --tags 'health-check or smoke' .
To cater for different purposes of validation the scenarios are tagged with one or more markers to state to which group(s) it belongs to. The selection is done through stating which group(s) to run. The available groups are presented next.
| Group | Purpose |
|---|---|
| health-check | Ensures the QA environment is properly set up and ready to perform the intended validations. |
| smoke | Performs a rapid validation using a small set of features and scenarios. Less time-consuming option to determine the QA status of a build. |
| coverage | Intended to validate the features and scenarios to a finer detail covering the most options and possible side effects. The most time-consuming operation to determine the QA status of a build. |
To build a beautified HTML report for a selection simply do:
cd docker
node ../tools/html_report.js -s </path/to/report/output.json> -o </path/to/pretty/report/output.html>Once the report is finished it outputs:
Cucumber HTML report /path/to/pretty/report/output.html generated successfully.
To view the beautified version of the report please open the /path/to/pretty/report/output.html file in a web browser.