feat(tests): added verifications for all the fields regarding positive_expected_result and actual results

[cx-ricardo-jesus]

Closes #

Reason for Proposed Changes

• Currently, there is no verification being made regarding the fields from the results, besides the filename, line and severity

Proposed Changes

• Using scripts that I will detail below, I filled all the positive_expected_results from each query.

• Also, changed the queries_test.go, more specifically the validateQueryResultFields which is the function that check's if the values from the results and the positve_expected_result from each query are equal. Basically, in this function I added verification's for the fields resourceType, resourceName, searchKey, searchValue, keyExpectedValue, keyActualValue and issueType. Also changed the function vulnerabilityCompare, to order in a runtime context, the results taking into account the fields from the expected results and actual results in the following order: fileName, line, searchKey, searchValue, resourceType, resourceName, queryName, keyExpectedResult, keyActualValue. This is to be sure that the actual results nd the expected results is being compared in the exact same order which did not happen before this change, since that if a field returns several results for the same line in the same file, it had the chance of returning an error stating that the expected and actual results are not equal even though that isn't true.

• The scripts that were created to automate the generation and population of the positive_expectedresults.json files for each KICS query, adding fields such as queryName, severity, line, filename, resourceType, resourceName, searchKey, searchValue, expectedValue, actualValue and issueType. Below I will explain each one of the scripts:

• The models.py, defines the data models used across all the scripts. Basically, the TestList structure has a list of QueryInfo structures. These QueryInfo structures has the fields test_path, results_file_path, id, payload_path, results_info, return_code and is_bom that will be filled by the generate.py and will be used by the runner.py to run KICS scan comands using these fields for each query. The ResultInfo, is used to store the results of each KICS scan.

• The main.py orchestrates the full pipeline of execution, the first step is to build the test list, using the models explained above and run the scan and populate the resultsInfo for each query with the results of every single scan. This will be made by using the function run_all inside the `runner.py.

• This run_all function will get the testList of all the queries using the build_test_list function from the generate.py script. After getting that testList with the QueryInfo fields with the information necessary to run the KICS scan commmands, it will run the scan for every single query sequentially.

• That helper function build_test_list(), defined inside the generate.py file, walks through the assets/queries and extract the necessary information to be included in the QueryInfo, to be later used in the runner.py script as mentioned above, to run the scan commands.

• After that the main.py, will use the function write_positive_expected_results from the write_expected_results.py file that basically iterates through the entire TestList structure and writes the information in each query respective positive_expected_result.json file inside the test/ directory.

• After running this, there are some queries that do not changed it's positive_expected__result.json file, because, there are no results, because the previous scripts only ran the scan command using the entire test directory at once. For those scenarios I created the script run_skipped.py that basically scans once per individual positive test file. In the main.py the information about all of those skipped queries, is made by using the write_skipped_queries_report from the write_expected_results.py file. The information of the skipped queries will be written in a json file.

• After that the main.py will run the run_skipped.py script to open that skipped queries reports written as explained above. In each one of those skipped queries, it will use the process_skipped_query function to run each positive file test individually instead of running the entire test directory at once. For each positive test it will run the individual scan using the run_individual_scan helper function that just runs the KICS scan command and returns the path for th results file and the returncode from the scan. After that it will run the parse_results_from_file function that parse that results file in json format and return a list of ResultInfo structure to be used

• After that there were some queries listed below that still got errors in the unit tests. After that I had to manually solve each case:
  

• The script add_issue_tyoe.py was created after realizing the issueType field had been omitted from the original scripts. It parses each query's .rego file to extract the issueType values, and adds only the issueType field to each existing query in the positive_expected_result.json files without modifying any other field. After running this script 40 queries had theyir issueType values incorrectly assigned due to the limitations of the static rego analysis (dynamic issueType resolution, among other scenarios). These queries, listed below, were manually corrected one by one.
  
  

I submit this contribution under the Apache-2.0 license.

[github-actions]

KICS version: v2.1.18

Category Results CRITICAL 0 HIGH 0 MEDIUM 0 LOW 0 INFO 0 TRACE 0 TOTAL 0	Metric Values Files scanned 1 Files parsed 1 Files failed to scan 0 Total executed queries 47 Queries failed to execute 0 Execution time 0