Automated Data Dictionary Testing Using the RESO Commander and IntelliJ
The RESO Commander is an automated, cross-platform testing tool written in Java that uses Cucumber BDD for Data Dictionary and Web API acceptance testing. IntelliJ has a plugin for running Cucumber tests, which provides a testing GUI that's easy to work with and also provides debugging support.
This walk-through provides information on how to configure your local environment to run the RESO Data Dictionary 1.7 tests that are used during certification, as well as how to create reports and interpret the results.
If you haven't done so already, clone the RESO Commander project to your local machine.
$ git clone https://github.com/RESOStandards/web-api-commander.git
Alternatively, you can clone the Commander project using the IntelliJ IDE, but it requires setting up GitHub in the IDE. It's often easier to do this from the command line.
Download IntelliJ here.
You can use the (free) Community Edition to run the tests. There are versions for Windows, MacOS, and Linux.
When you first start IntelliJ, it will ask whether you want to create a new project, open an existing project, or check out code from a repository.
Choose Open or Import and select the folder that you downloaded the source code to.
You will need a Java JDK installed to run the tests. Here are instructions for how to set it up in IntelliJ. You only need to follow step 3 in the section linked to. Open JDK is recommended.
In order to run Cucumber tests, you'll need to install plugins.
In the IntelliJ IDE, go to File > Settings and navigate to the Plugins option, then search for Cucumber in the Marketplace:
Install the Cucumber for Java plugin. When you do this, you will be shown the following screen asking you to install Gherkin as well:
Select Install to proceed.
Now that IntelliJ and Cucumber have been installed, you'll need to create a configuration for each separate certification you'd like to run. Many people will only need one Data Dictionary configuration, but if you have more than one you can copy the existing configs and create new ones.
To create your first Data Dictionary 1.7 configuration, locate the configuration menu in IntelliJ. This should be along the top bar in the IDE, and will look similar to the following:
The screenshot above shows an existing configuration called "commander." If this is the first time you're using IntelliJ on your local machine, your options may be slightly different.
Select the menu and you should see an Edit Configurations option:
Select the Edit Configurations option, and something similar to the following will appear:
Select the + option and a configuration screen similar to the following will appear:
Enter the following values:
| Setting | Value |
|---|---|
| Main Class | io.cucumber.core.cli.Main |
| Glue | org.reso.commander.test.stepdefs |
| Feature or Folder Path | /path/to/web-api-commander/src/test/java/org/reso/commander/test/features |
| VM Options | -DpathToRESOScript=/path/to/your/data-dictionary-1.7.resoscript -DshowResponses=true |
| Program Arguments | --plugin org.jetbrains.plugins.cucumber.java.run.CucumberJvm4SMFormatter --strict |
| Working Directory | (leave this blank) |
| Environment Variables | (leave this blank) |
| Use classpath of module | web-api-commander.main |
| JRE | Default (should be 1.8, 10, or 11) |
RESOScript Files: the -DpathToRESOScript should point to a RESOScript file containing the configuration for your server. This file contains credentials, which can be either an OAuth2 token or Client Credentials and a server URL. A template file exists here.
Once you have entered these values, press OK to save the configuration. You should be able to run the tests at this point.
There should be a green "play" arrow in your IntelliJ environment, usually near the top right of the IDE window. This is how to start the tests.
The environment will build the necessary code and run the tests:
The testing tool will output relevant messages at each important step, or when there are error messages. If the tests pass, the output will look similar to the following:
You can expand each of the steps, and right click and run or debug each one individually. In Data Dictionary terminology, a Resource is a Cucumber Feature, and a Field is a Scenario. Each Scenario has many steps.
The Data Dictionary Testing Specification outlines exactly what's being tested, but generally it's the following:
- Make a metadata request using an OAuth2 token or client credentials.
- Validate the XML response using OASIS-published XSDs to ensure it's in the OData XML Metadata format, that it conforms to naming conventions and special character restrictions, that each resource has a key and that each resource is well-defined, and checking to see that any other declared types or local features a provider uses pass OData compliance rules.
There are additional IntelliJ tools that can be useful if OData validation fails. The OASIS XSDs are committed to the GitHub repository so that validation may be done locally.
The local XSD files from OASIS are shown in the following screenshot. They can be found in the path src/main/resources relative to the root of the project, and are called edm.4.0.errata03.xsd and edmx.4.0.errata03.xsd. The first time you try and open an XML Metadata file, you will be prompted to resolve the referenced schema. Point IntelliJ to these files if it doesn't find them automatically.
IntelliJ uses the OASIS XML Schema Definition files, and will raise issues when there are syntax errors in the XML Metadata. In the following example, a space was added in the enumeration name "Door Features" which is invalid in OData and causes an error:
You will also notice that TODOs have been generated in lookups without any enumerations. If you use the RESO reference metadata for your own server and don't have these lookups defined, you should remove them as the keys are made up placeholders to conform to OData's validation rules while still providing a placeholder for the underlying field.
Here is a summary of the Data Dictionary 1.7 testing rules:
- At least one of the supported resources in the Data Dictionary MUST be present. This means Property, Member, Office, etc. See: RESO Data Dictionary 1.7 Wiki.
- For each resource, the data types of each field are checked to make sure they match what's in the Dictionary. This is part of RCP-031 in the RESO Web API.
- For RESO standard fields, the system checks if synonyms (disallowed) are present. If so, the tests will fail.
The primary goal of Data Dictionary 1.7 Certification is to ensure OData compliance and type mappings are correct for anything that was found, which ensures interoperability and compatibility with off-the-shelf tools and libraries, and to create statistics to help drive adoption and provide information for data consumers.
There are additional reports provided by a Gradle task that uses the Cucumber Reporting plugin. See build/certification/reports.
