Merge pull request #341 from iriusrisk/release/1.20.0

Release/1.20.0 to main
iriusrisk · Nov 6, 2023 · 1a30521 · 1a30521
2 parents 584f067 + 77ad228
commit 1a30521
Show file tree

Hide file tree

Showing 231 changed files with 8,643 additions and 3,596 deletions.
diff --git a/README.md b/README.md
@@ -1,8 +1,9 @@
 # StartLeft
 
+![Supported python versions](https://img.shields.io/badge/python-3.8%20%7C%203.9%20%7C%203.10%20%7C%203.11-blue)
+![Software Quality Assurance](https://github.com/iriusrisk/startleft/actions/workflows/qa.yml/badge.svg)
 ![Semgrep scan with owasp-top-ten & cwe-top-25](https://github.com/iriusrisk/startleft/actions/workflows/semgrep.yml/badge.svg)
 ![Documentation](https://github.com/iriusrisk/startleft/actions/workflows/documentation.yml/badge.svg)
-![Software Quality Assurance](https://github.com/iriusrisk/startleft/actions/workflows/qa.yml/badge.svg)
 
 **StartLeft** is an automation tool for **generating Threat Models** written in the 
 [Open Threat Model (OTM)](http://iriusrisk.github.io/startleft/site/Open-Threat-Model-%28OTM%29/)

diff --git a/_sl_build/modules.py b/_sl_build/modules.py
@@ -4,7 +4,8 @@
 
 STARTLEFT_MODULE = {'name': 'startleft', 'type': 'general', 'allowed_imports': ['slp_base', 'otm', 'sl_util']}
 # TODO Startleft needs to depend on TF and CFT processors until a decision is token about the search function
-STARTLEFT_MODULE['allowed_imports'].extend(['slp_cft', 'slp_tf'])
+# TODO Startleft needs to depend on VISIO processors until a decision is token about the summary function
+STARTLEFT_MODULE['allowed_imports'].extend(['slp_cft', 'slp_tf', 'slp_visio'])
 
 # TODO Dependency between otm and sl_util must be removed
 OTM_MODULE = {'name': 'otm', 'type': 'general', 'allowed_imports': ['sl_util']}
@@ -19,7 +20,9 @@
     {'name': 'slp_tfplan', 'type': 'processor', 'provider_type': 'TFPLAN', 'allowed_imports': _slp_allowed_imports},
     {'name': 'slp_visio', 'type': 'processor', 'provider_type': 'VISIO', 'allowed_imports': _slp_allowed_imports},
     {'name': 'slp_visio', 'type': 'processor', 'provider_type': 'LUCID', 'allowed_imports': _slp_allowed_imports},
-    {'name': 'slp_mtmt', 'type': 'processor', 'provider_type': 'MTMT', 'allowed_imports': _slp_allowed_imports}
+    {'name': 'slp_mtmt', 'type': 'processor', 'provider_type': 'MTMT', 'allowed_imports': _slp_allowed_imports},
+    # TODO Set type to processor to make the endpoint available
+    {'name': 'slp_drawio', 'provider_type': 'DRAWIO', 'allowed_imports': _slp_allowed_imports}
 ]
 
 """

diff --git a/deployment/Dockerfile.application b/deployment/Dockerfile.application
@@ -1,4 +1,4 @@
-FROM python:3.8-alpine
+FROM python:3.8-alpine AS startleft-base
 
 WORKDIR /usr/src/app
 
@@ -15,9 +15,21 @@ COPY . .
 
 RUN pip install .
 
-RUN rm -r ../app/*
 
-# Remove not needed dependencies for runtime
-RUN apk del git geos-dev re2-dev py3-pybind11-dev
+FROM python:3.8-alpine
+
+WORKDIR /app
+
+RUN apk update && \
+    apk add libmagic && \
+    apk add re2 && \
+    apk add graphviz && \
+    apk add lapack && \
+    apk add cblas && \
+    apk add geos
+
+COPY --from=startleft-base /usr/local/lib/python3.8/site-packages /usr/local/lib/python3.8/site-packages
+
+COPY --from=startleft-base /usr/local/bin/startleft /usr/local/bin/startleft
 
 CMD ["startleft", "server", "--host", "0.0.0.0"]
diff --git a/docs/Quickstart-Guide-for-Beginners.md b/docs/Quickstart-Guide-for-Beginners.md
@@ -5,7 +5,7 @@ some commands, set up the REST API and, in summary, familiarize yourself with th
 ## Prerequisites
 
 ---
-* Install the **[latest version of Python](https://www.python.org/downloads/)**.
+* Install a **[Python version between 3.8 and 3.11](https://www.python.org/downloads/)**.
 * Install **[pip3](https://pip.pypa.io/en/stable/installation/)**.
 * Install **[git](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git).**
 * Install **[graphviz and graphviz-dev](https://pygraphviz.github.io/documentation/stable/install.html#ubuntu-and-debian).**

diff --git a/docs/integration/Quickstart-Guide-for-Integrations.md b/docs/integration/Quickstart-Guide-for-Integrations.md
@@ -59,7 +59,7 @@ git checkout release/1.5.0
 
 Now, we can create the StartLeft image:
 ```shell
-docker build . -f deployment/Dockerfile.docs.application.application -t startleft
+docker build . -f deployment/Dockerfile.application -t startleft
 ```
 
 And, finally, we can run the docker container for the image we have just generated. Notice that you can select the

diff --git a/docs/startleft-processors/diagram/Lucidchart-support.md b/docs/startleft-processors/diagram/Lucidchart-support.md
@@ -28,6 +28,25 @@ About the parsing logic:
 * Dataflows are calculated based on their position, what means that they do not necessarily need to _touch_ origin 
   or target shapes, but they have some tolerance.
 
+## Catch All Configuration
+This processor includes an exclusive functionality to activate the mapping for all the shapes not included in the components' mapper section.
+All the unknown shapes will be mapped to the type defined under the `catch_all` property
+```yaml
+configuration:
+  catch_all: empty-component
+```
+
+#### Skip
+This configuration defines a list of resources that will never be mapped.
+```yaml
+configuration:
+  skip:
+    - AmazonCloudWatch
+    - AmazonDynamoDB
+    - AmazonRoute53
+    - DataSharesAzure2021
+```
+
 ## An example
 
 In this example, we can see a Lucidchart diagram which includes different types of elements.
@@ -68,8 +87,6 @@ Then, we can map the generic shapes by name in a custom mapping file:
         
       - label: Android
         type: android-device-client
-
-    dataflows: []
     ```
 
 The expected result for this case should be an OTM like this:
@@ -254,7 +271,7 @@ curl --location --request POST localhost:5000/api/v1/startleft/diagram \
 ### Command line usage
 You can also use the Command Line option for this example, with the files downloaded in the previous section.
 
-Make sure StartLeft is [properly installed](../../../Quickstart-Guide-for-Beginners/#install-startleft) 
+Make sure StartLeft is [properly installed](../../Quickstart-Guide-for-Beginners.md#install-startleft) 
 and execute the following command:
 
 ```shell

diff --git a/docs/startleft-processors/diagram/Visio-Mapping.md b/docs/startleft-processors/diagram/Visio-Mapping.md
@@ -55,10 +55,11 @@ arrays for the mappings of each type of element:
 ```yaml
 trustzones: []
 components: []
-dataflows: []
+configuration: {}
 ```
 
-Each of these arrays contains the information for mapping shapes into TrustZones, Components or Dataflows, respectively. 
+Each of these arrays contains the information for mapping shapes into TrustZones, Components, or configure 
+particular behavior for the processor.  
 Also note that all three are mandatory and have to be included in each mapping file, even if they only contain an empty array.
 
 ### Mapping TrustZones 
@@ -353,8 +354,7 @@ The resulting OTM will be as follows:
     More info in the <a href="https://learn.microsoft.com/en-us/openspecs/sharepoint_protocols/ms-vsdx/e58f5f25-76d8-4f65-ae24-d286b10168d7" target="_blank">official Microsoft documentation</a>.
 
 ### Mapping DataFlows
-Despite the fact that a `dataflows` tag is already defined in the mapping file structure, and it is required by the schema,
-the DataFlows mapping process is fixed and not configurable. Basically, it takes all the arrows in the Visio source that connect 
+The DataFlows mapping process is fixed and not configurable. Basically, it takes all the arrows in the Visio source that connect 
 components that are mapped and create a DataFlow for them. If some arrow connects shapes that are not mapped, the 
 DataFlow is not created. This can be easily understood with the following picture:
 
@@ -388,7 +388,7 @@ All these functionalities are available to map both, components and TrustZones.
 
     This configuration sets all the shapes of name/type `AmazonS3` or `AmazonSimpleStorageServiceS3` to components of type `s3`
 
-#### Mapping by a Regex
+### Mapping by a Regex
 
 ```yaml
   - label: {$regex: ^AWS Region:.*$}

diff --git a/docs/startleft-processors/diagram/Visio-Quickstart.md b/docs/startleft-processors/diagram/Visio-Quickstart.md
@@ -100,8 +100,6 @@ components:
 
   - label:  Database
     type:   rds
-
-dataflows: []
 ```
 
 On the other hand, for this specific request, you need to provide a **custom mapping file** which contains the 
@@ -119,8 +117,6 @@ components:
 
   - label:  My Custom VPC
     type:   empty-component
-
-dataflows: []
 ```
 
 The result of sending to StartLeft this diagram with these mapping files would be an OTM with all the components we 

diff --git a/docs/startleft-processors/diagram/Visio-Summary.md b/docs/startleft-processors/diagram/Visio-Summary.md
@@ -0,0 +1,104 @@
+# What is Visio Summary?
+
+---
+
+> Visio Summary is a tool available at our Command Line Interface for retrieving useful info from VSDX files.
+
+This tool can retrieve all the shapes' information available (type, name) 
+and their candidate OTM type by emulating the parse method. 
+
+You can find [here](../../usage/Command-Line-Interface.md#summary) a complete explanation of this CLI function.
+
+## Summary Options
+This summary tool can be executed with multiple configurations:
+
+
+### without mapping file
+---
+!!! note ""
+
+    The summary retrieves all the availables shapes in the VSDX files without their candidate OTM type.
+
+=== "CLI execution"
+    ```shell
+    startleft summary \
+    --diagram-type VISIO \
+    examples/visio/aws-with-tz-and-vpc.vsdx
+    ```
+
+### by `file path`
+---
+!!! note ""
+
+    The summary is executed against a unique Visio file.
+
+=== "CLI execution"
+    ```shell
+    startleft summary \
+    --diagram-type VISIO \
+    --default-mapping-file examples/visio/iriusrisk-visio-aws-mapping.yaml \
+    examples/visio/aws-with-tz-and-vpc.vsdx
+    ```
+
+### by `multiple file path`
+---
+!!! note ""
+
+    The summary is executed against multiple Visio files. 
+
+=== "CLI execution"
+    ```shell
+    startleft summary \
+    --diagram-type VISIO \
+    --default-mapping-file examples/visio/iriusrisk-visio-aws-mapping.yaml \
+    examples/visio/aws-with-tz-and-vpc.vsdx examples/visio/visio-basic-example.vsdx
+    ```
+
+### by `folder path`
+---
+!!! note ""
+
+    The summary is executed against a folder path that contains `.vsdx` in it. 
+
+=== "CLI execution"
+    ```shell
+    startleft summary \
+    --diagram-type VISIO \
+    --default-mapping-file examples/visio/iriusrisk-visio-aws-mapping.yaml \
+    examples/visio/
+    ```
+
+### by `multiple folder path`
+---
+!!! note ""
+
+    The summary is executed against multiple folder path that contains `.vsdx` in it. 
+
+=== "CLI execution"
+    ```shell
+    startleft summary \
+    --diagram-type VISIO \
+    --default-mapping-file examples/visio/iriusrisk-visio-aws-mapping.yaml \
+    examples/visio/folder1 examples/visio/folder2
+    ```
+
+## Summary Output Example
+
+```
+| SOURCE      | SOURCE_ELEMENT_TYPE | SOURCE_ELEMENT_NAME   | OTM_MAPPED_TYPE |
+|-------------|---------------------|-----------------------|-----------------|
+| file_1.vsdx |                     | Public Cloud          |                 |
+| file_1.vsdx |                     | Custom VPC            |                 |
+| file_1.vsdx |                     | Private Secured Cloud |                 |
+| file_1.vsdx | Amazon CloudWatch   | Amazon CloudWatch     | cloudwatch      |
+| file_1.vsdx | Amazon CloudWatch   | Custom log system     | cloudwatch      |
+| file_1.vsdx | Amazon EC2          | Amazon EC2            | ec2             |
+| file_1.vsdx | Amazon EC2          | Custom machine        | ec2             |
+| file_1.vsdx | Database            | Private Database      | rds             |
+| file_2.vsdx |                     | Private Secured Cloud |                 |
+| file_2.vsdx |                     | Public Cloud          |                 |
+| file_2.vsdx |                     | My Custom VPC         |                 |
+| file_2.vsdx |                     | My Custom Machine     |                 |
+| file_2.vsdx | Amazon EC2          | My EC2                | ec2             |
+| file_2.vsdx | Database            | Private Database      | rds             |
+```
diff --git a/docs/usage/Command-Line-Interface.md b/docs/usage/Command-Line-Interface.md
@@ -33,6 +33,7 @@ Commands:
   parse     Parses source files into Open Threat Model
   search    Searches source files for the given query
   server    Launches the REST server to generate OTMs from requests
+  summary   Generates a summary CSV file of the given source files
   validate  Validates a mapping or OTM file
 
 ```
@@ -89,6 +90,7 @@ The list of commands that can be used to work in CLI mode is detailed as follows
 | validate | Validates a mapping or OTM file.                         |
 | search   | Searches source files for the given query.               |
 | server   | Launches the REST server to generate OTMs from requests. |
+| summary  | Generate a summary CSV file.                             |
 
 
 
@@ -777,4 +779,64 @@ receive one or more IaC files, process them and give back the OTM file in the re
     INFO    on - Waiting for application startup.
     INFO    on - Application startup complete.
     INFO    server - Uvicorn running on http://127.0.0.1:5000 (Press CTRL+C to quit)
-    ```
+    ```
+
+### Summary
+
+This command **(only available for VISIO/LUCID)** returns a summary CSV which contains
+all the source elements available and their candidate OTM type by emulating the parse method.
+
+The CSV contains the following info:
+
+  - SOURCE: The source file name
+  - SOURCE_ELEMENT_TYPE: The type of the element in the source
+  - SOURCE_ELEMENT_NAME: The name of the element in the source
+  - OTM_MAPPED_TYPE: The type of the element in the OTM
+
+```shell
+Usage: startleft summary [OPTIONS] [SOURCE_FILES]...
+
+  Generates a summary CSV file of the given source files
+
+Options:
+  -g, --diagram-type [VISIO|LUCID]
+                                  The diagram file type.  [required]
+  -d, --default-mapping-file TEXT
+                                  Default mapping file to parse the diagram
+                                  file.
+  -c, --custom-mapping-file TEXT  Custom mapping file to parse the diagram
+                                  file.
+  -o, --output-file TEXT          Summary output file.
+  --help                          Show this message and exit.
+```
+
+??? example "`Lucid` example"
+
+    === "CLI execution"
+        ```shell
+        startleft summary \
+        --diagram-type LUCID \
+        --default-mapping-file examples/lucidchart/iriusrisk-lucid-aws-mapping.yaml \
+        examples/lucidchart/lucid-aws-with-tz-and-vpc.vsdx
+        ```
+
+    === "summary.csv"
+        ```csv
+        | SOURCE                         | SOURCE_ELEMENT_TYPE          | SOURCE_ELEMENT_NAME            | OTM_MAPPED_TYPE           |
+        |--------------------------------|------------------------------|--------------------------------|---------------------------|
+        | lucid-aws-with-tz-and-vpc.vsdx | AWSCloud                     | Public Cloud                   | empty-component           |
+        | lucid-aws-with-tz-and-vpc.vsdx | AWSCloudTrail                | My CloudTrail                  | cloudtrail                |
+        | lucid-aws-with-tz-and-vpc.vsdx | AmazonAPIGateway_purple      | My API Gateway                 | api-gateway               |
+        | lucid-aws-with-tz-and-vpc.vsdx | AmazonCloudWatch             | My CloudWatch                  | cloudwatch                |
+        | lucid-aws-with-tz-and-vpc.vsdx | AmazonEC2                    | My EC2                         | ec2                       |
+        | lucid-aws-with-tz-and-vpc.vsdx | AmazonSimpleStorageServiceS3 | My Simple Storage Service (S3) | s3                        |
+        | lucid-aws-with-tz-and-vpc.vsdx | Client                       | Web browser                    | generic-client            |
+        | lucid-aws-with-tz-and-vpc.vsdx | DatabaseBlock                | My DynamoDB                    | other-database            |
+        | lucid-aws-with-tz-and-vpc.vsdx | DefaultSquareBlock           | Custom VPC                     | empty-component           |
+        | lucid-aws-with-tz-and-vpc.vsdx | DefaultSquareBlock           | Internet                       | empty-component           |
+        | lucid-aws-with-tz-and-vpc.vsdx | Mobileclient                 | Android                        |                           |
+        | lucid-aws-with-tz-and-vpc.vsdx | RectangleBlock               | Private Secured Cloud          |                           |
+        | lucid-aws-with-tz-and-vpc.vsdx | SQLDatabaseAzure2021         | SQL Database                   | CD-MICROSOFT-AZURE-SQL-DB |
+        ```
+
+
diff --git a/examples/lucidchart/iriusrisk-lucid-aws-mapping.yaml b/examples/lucidchart/iriusrisk-lucid-aws-mapping.yaml
@@ -368,4 +368,8 @@ components:
     type: vpc
   - label: { $regex: "^(AmazonEC2_?|EC2)a-zA-Z?[0-9]?a-z?(Instance|instance)s?$" }
     type: ec2
-dataflows: [ ]
+
+dataflows: [ ]
+
+configuration:
+  catch_all: empty-component
diff --git a/examples/visio/iriusrisk-visio-aws-mapping.yaml b/examples/visio/iriusrisk-visio-aws-mapping.yaml
@@ -1678,5 +1678,3 @@ components:
 
   - label: VPC Access Points
     type: empty-component
-
-dataflows: []
diff --git a/mkdocs.yml b/mkdocs.yml
@@ -47,6 +47,7 @@ nav:
           - Visio Mapping: startleft-processors/diagram/Visio-Mapping.md
           - Visio TrustZones Mapping: startleft-processors/diagram/Visio-TrustZones-Mapping.md
           - Visio Representations: startleft-processors/diagram/Visio-Representations.md
+          - Visio Summary: startleft-processors/diagram/Visio-Summary.md
           - Lucidchart Support: startleft-processors/diagram/Lucidchart-support.md
   - Troubleshooting: Troubleshooting.md
Original file line number	Diff line number	Diff line change
Expand Up		@@ -1678,5 +1678,3 @@ components:

		- label: VPC Access Points
		type: empty-component

		dataflows: []