Merge branch 'dev' of https://github.com/CDOT-CV/jpo-ode into missing…

…-ingest-log-alerts

.devcontainer/Dockerfile

@@ -74,4 +74,7 @@ RUN apt-get install -y g++ cmake libsasl2-dev libsasl2-modules libssl-dev librdk
 RUN apt install -y autoconf libtool
 
 ENV REDACTION_PROPERTIES_PATH /workspaces/jpo-ode/jpo-cvdp/config/fieldsToRedact.txt
-ENV RPM_DEBUG true
+ENV RPM_DEBUG true
+
+# Python3
+RUN apt-get install -y python3 python3-venv python3-pip

.devcontainer/devcontainer.json

@@ -26,7 +26,8 @@
 				"ms-vscode.cpptools",
 				"ms-vscode.cmake-tools",
 				"ms-vscode.makefile-tools",
-				"Oracle.oracle-java"
+				"Oracle.oracle-java",
+				"ms-python.python"
 			]
 		}
 	},

Makefile

@@ -7,6 +7,7 @@ default:
 	$(info `make rebuild` to stop, delete, and then rebuild the containers)
 	$(info `make clean-build` to rebuild the containers without using the cache)
 
+.PHONY: start
 start:
 ifeq ("$(wildcard .env)", "")
 	$(error "ERROR: jpo-ode Environment file `.env` not found in ${PWD}")
@@ -25,14 +26,18 @@ ifeq ("$(wildcard ./jpo-utils/.env)", "")
 endif
 	docker compose build
 
+.PHONY: stop
 stop:
 	docker compose down
 
+.PHONY: delete
 delete:
 	docker compose down -v
 
+.PHONY: rebuild
 rebuild:
 	$(MAKE) stop delete build start
 
+.PHONY: clean-build
 clean-build:
 	docker compose build --no-cache

README.md

@@ -30,22 +30,22 @@ All stakeholders are invited to provide input to these documents. To provide fee
 
 The current version and release history of the JPO-ODE: [ODE Release Notes](<docs/Release_notes.md>)
 
-**Table of Contents**
-
-1. [Usage Example](#usage-example)
-1. [Configuration](#configuration)
-1. [Installation](#installation)
-1. [File Manifest](#file-manifest)
-1. [Development Setup](#development-setup)
-1. [Release History](#release-history)
-1. [Contact Information](#contact-information)
-1. [Contributing](#contributing)
-1. [Credits and Acknowledgement](#credits-and-acknowledgement)
-1. [Code.gov Registration Info](#codegov-registration-info)
-1. [Kubernetes](#kubernetes)
-1. [Sonar Cloud](#sonar-token-configuration) ([Documentation](https://sonarcloud.io/documentation/user-guide/user-token/))
-1. [SNMP](#snmp)
-1. [GitHub Artifact Usage](#githhub-artifact-usage)
+## Table of Contents
+
+1. [Usage Example](#1-usage-example)
+1. [Configuration](#2-configuration)
+1. [Installation](#3-installation)
+1. [File Manifest](#4-file-manifest)
+1. [Development Setup](#5-development-setup)
+1. [Release History](#6-release-history)
+1. [Contact Information](#7-contact-information)
+1. [Contributing](#8-contributing)
+1. [Credits and Acknowledgement](#9-credits-and-acknowledgement)
+1. [Code.gov Registration Info](#10-codegov-registration-info)
+1. [Kubernetes](#11-kubernetes)
+1. [Sonar Cloud](#12-sonar-token-configuration) ([Documentation](https://sonarcloud.io/documentation/user-guide/user-token/))
+1. [SNMP](#13-snmp)
+1. [GitHub Artifact Usage](#14-gitHub-artifact-usage)
 
 <!--
 #########################################
@@ -104,7 +104,7 @@ Supported message types:
   <img src="./docs/images/readme/figure3.png" width="80%" height="50%">
 </p>
 
-[Back to top](#toc)
+[Back to top](#table-of-contents)
 
 
 
@@ -173,7 +173,7 @@ ODE configuration can be customized for every deployment environment using envir
 
 You must rename `sample.env` to `.env` for Docker to automatically read the file. This file will contain AWS access keys and other private information. Do not push this file to source control.
 
-[Back to top](#toc)
+[Back to top](#table-of-contents)
 
 
 
@@ -217,6 +217,12 @@ You may download the stable, default branch for ALL of these dependencies by usi
 git clone --recurse-submodules https://github.com/usdot-jpo-ode/jpo-ode.git
 ```
 
+If you have already cloned the repository, you can use the following command to download the stable, default branch for all dependencies by using the following command:
+
+```bash
+git submodule update --init --recursive
+```
+
 Once you have these repositories obtained, you are ready to build and deploy the application.
 
 ##### Downloading the source code from a non-default branch
@@ -279,15 +285,11 @@ git submodule deinit -f . && git submodule update --recursive --init
 
 - Docker builds may fail if you are on a corporate network due to DNS resolution errors.
 [See here](https://github.com/usdot-jpo-ode/jpo-ode/wiki/Docker-fix-for-SSL-issues-due-to-corporate-network) for instructions to fix this.
-- In order for Docker to automatically read the environment variable file, you must rename it from `sample.env` to `.env`. **This file will contain private keys, do not put add it to version control.**
-
-Copy the following files from `jpo-ode` directory into your DOCKER_SHARED_VOLUME directory.
-- Copy jpo-ode/ppm.properties to ${DOCKER_SHARED_VOLUME}/config.properties. Open the newly copied `config.properties` file in a text editor and update the `metadata.broker.list=your.docker.host.ip:9092` line with your system's DOCKER_HOST_IP in place of the dummy `your.docker.host.ip` string.
-- Copy jpo-ode/adm.properties to ${DOCKER_SHARED_VOLUME}/adm.properties
-- Copy jpo-ode/aem.properties to ${DOCKER_SHARED_VOLUME}/aem.properties
-- Copy jpo-utils/sample.env to jpo-utils/.env
-  - Fill in the variables as described in the [README](jpo-utils/README.md)
-- If you want to see log-based alerts notifying you if no TIMs were ingested in a specific period of time, you will want to update your `.env` file to set `ODE_TIM_INGEST_MONITORING_ENABLED=true` and `ODE_TIM_INGEST_MONITORING_INTERVAL=<seconds between ingest count checks>`. See [TimIngestWatcher](jpo-ode-svcs/src/main/java/us/dot/its/jpo/ode/traveler/TimIngestWatcher.java) to see the log-based monitoring provided.
+- In order for Docker to automatically read the environment variable files, you must:
+  - Make a copy of [sample.env](./sample.env) rename it as `.env` **_and_**;
+  - Make a copy of [jpo-utils/sample.env](jpo-utils/sample.env), rename it as `.env` (keep this one in the `jpo-utils/` directory), and fill in the variables as described in the [jpo-utils README](jpo-utils/README.md)
+  - **The .env files will contain private keys, do not add them to version control.**
+  - If you want to see log-based alerts notifying you if no TIMs were ingested in a specific period of time, you will want to update your `.env` file to set `ODE_TIM_INGEST_MONITORING_ENABLED=true` and `ODE_TIM_INGEST_MONITORING_INTERVAL=<seconds between ingest count checks>`. See [TimIngestWatcher](jpo-ode-svcs/src/main/java/us/dot/its/jpo/ode/traveler/TimIngestWatcher.java) to see the log-based monitoring provided.
 
 **Make:**
 
@@ -319,20 +321,39 @@ Makefile:14: *** "ERROR: jpo-utils Environment file `.env` not found in ".  Stop
 
 **Docker Compose:**
 
-Navigate to the root directory of the jpo-ode project and run the following command:
+Navigate to the root directory of the jpo-ode project and run:
+
+```bash
+make start
+```
+
+OR
 
 ```bash
 docker compose up --build -d
 docker compose ps
 ```
 
-To bring down the services and remove the running containers run the following command:
+To bring down the services and remove the running containers run:
+
+```bash
+make stop
+```
+
+OR
 
 ```bash
 docker compose down
 ```
+
 For a fresh restart, run:
 
+```bash
+make rebuild
+```
+
+OR
+
 ```bash
 docker compose down
 docker compose up --build -d
@@ -341,9 +362,15 @@ docker compose ps
 
 To completely rebuild from scratch, run:
 
+```bash
+make rebuild
+```
+
+OR
+
 ```bash
 docker compose down
-docker compose rm -fvs
+docker compose rm -v
 docker compose up --build -d
 docker compose ps
 ```
@@ -361,6 +388,7 @@ To configure what services are started, use the `COMPOSE_PROFILE` environmental
 Profiles are also available for each service name to individually specify a service to enable.
 
 #### asn1_codec Module (ASN.1 Encoder and Decoder)
+
 ODE requires the deployment of asn1_codec module. ODE's `docker-compose.yml` file is set up to build and deploy the module in a Docker container. If you wish to run `asn1_codec` module outside Docker (i.e. directly on the host machine), please refer to the documentation of `asn1_codec` module.
 
 The only requirement for deploying `asn1_codec` module on Docker is the setup of two environment variables `DOCKER_HOST_IP` and `DOCKER_SHARED_VOLUME`.
@@ -382,7 +410,7 @@ During the build process, edit the sample config file located in `config/example
 
 After a successful build, use the following commands to configure and run the PPM
 
-```
+```bash
 cd $BASE_PPM_DIR/jpo-cvdp/build
 $ ./bsmjson_privacy -c ../config/ppm.properties
 ```
@@ -417,7 +445,7 @@ In order to utilize Confluent Cloud:
 
 This has only been tested with Confluent Cloud but technically all SASL authenticated Kafka brokers can be reached using this method.
 
-[Back to top](#toc)
+[Back to top](#table-of-contents)
 
 # MongoDB Integration
 
@@ -439,7 +467,7 @@ For further documentation on configuring the MongoDB Kafka Connect image refer [
 
 Kafka connect is being used for MongoDB in this implementation but it can interact with many types of databases, here is further documentation for [kafka connect](https://docs.confluent.io/platform/current/connect/index.html)
 
-[Back to top](#toc)
+[Back to top](#table-of-contents)
 
 <!--
 #########################################
@@ -488,7 +516,7 @@ This section outlines the software technology stacks of the ODE.
 - [Stomp Websocket](http://jmesnil.net/stomp-websocket)
 - [SockJS](https://github.com/sockjs)
 
-[Back to top](#toc)
+[Back to top](#table-of-contents)
 
 
 
@@ -511,14 +539,11 @@ Install the IDE of your choice:
 * IntelliJ: [https://www.jetbrains.com/idea/](https://www.jetbrains.com/idea/)
 * VSCode: [https://code.visualstudio.com/](https://code.visualstudio.com/)
 
-### Continuous Integration
-
-See the [GitHub Workflows](.github/workflows/) defined for this project.
-
 ### Dev Container Environment
+
 The project can be reopened inside of a dev container in VSCode. This environment should have all of the necessary dependencies to debug the ODE and its submodules. When attempting to run scripts in this environment, it may be necessary to make them executable with "chmod +x" first.
 
-[Back to top](#toc)
+[Back to top](#table-of-contents)
 
 
 
@@ -534,7 +559,7 @@ The project can be reopened inside of a dev container in VSCode. This environmen
 
 [Release Notes](ReleaseNotes.md)
 
-[Back to top](#toc)
+[Back to top](#table-of-contents)
 
 
 
@@ -561,7 +586,7 @@ Unless required by applicable law or agreed to in writing, software distributed
 the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either expressed or implied. See the License for the specific language governing
 permissions and limitations under the [License](http://www.apache.org/licenses/LICENSE-2.0).
 
-[Back to top](#toc)
+[Back to top](#table-of-contents)
 
 
 
@@ -602,15 +627,15 @@ Code quality assurance is reported through the [usdot-jpo-ode SonarCloud organiz
 
 For regression and user acceptance testing, ODE provides an automated test harness. The test harness is provided in the [qa/test-harness](ga/test-harness) directory under jpo-ode root folder. The test harness uses the ODE [Validator Library](https://github.com/usdot-jpo-ode/ode-output-validator-library) repository as a submodule.
 
-For more information, please see: https://github.com/usdot-jpo-ode/jpo-ode/wiki/Using-the-ODE-test-harness
+For more information, please see: <https://github.com/usdot-jpo-ode/jpo-ode/wiki/Using-the-ODE-test-harness>
 
 ### Troubleshooting
 
 Please read our [Wiki](https://github.com/usdot-jpo-ode/jpo-ode/wiki) for more information, or check the [User Guide](docs/UserGuide.md).
 
 Application Support for the ODE currently managed via GitHub's native issue tracker: <https://github.com/usdot-jpo-ode/jpo-ode/issues>.
 
-[Back to top](#toc)
+[Back to top](#table-of-contents)
 
 
 
@@ -626,7 +651,7 @@ Application Support for the ODE currently managed via GitHub's native issue trac
 
 [Attribution](ATTRIBUTION.md)
 
-[Back to top](#toc)
+[Back to top](#table-of-contents)
 
 
 
@@ -666,7 +691,7 @@ Contact Phone: (202) 366-3000
 The ODE can be run in a Kubernetes (k8s) environment.
 See [the Kubernetes document](./docs/Kubernetes.md) for more details about this.
 
-[Back to top](#toc)
+[Back to top](#table-of-contents)
 
 
 <!--
@@ -678,16 +703,19 @@ See [the Kubernetes document](./docs/Kubernetes.md) for more details about this.
 <a name="sonar-token-configuration"></a>
 
 ## 12. Sonar Token Configuration
+
 Generating and Using Tokens
 Users can generate tokens that can be used to run analyses or invoke web services without access to the user's actual credentials.
 
 USDOT-JPO-ODE SonarCloud Organization : https://sonarcloud.io/organizations/usdot-jpo-ode-1/
 
 ### Generating a token
+
 You can generate new tokens at User > My Account > Security.
 The form at the bottom of the page allows you to generate new tokens. Once you click the Generate button, you will see the token value. Copy it immediately; once you dismiss the notification you will not be able to retrieve it.
 
 ### Using a token
+
 SonarScanners running in GitHub Actions can automatically detect branches and pull requests being built so you don't need to specifically pass them as parameters to the scanner.
 
 **<ins>To analyze your projects with GitHub Actions, you need to: </ins>**
@@ -706,9 +734,10 @@ Configure your workflow YAML file as below:
 Commit and push your code to start the analysis.
 
 ### Revoking a token
+
 You can revoke an existing token at User > My Account > Security by clicking the Revoke button next to the token.
 
-[Back to top](#toc)
+[Back to top](#table-of-contents)
 
 
 <!--
@@ -811,6 +840,4 @@ Finally, set the environment variables:
 * PACKAGE_READ_USERNAME - User name with read access to the repositories containing the packages.
 * PACKAGE_READ_TOKEN - Personal access token with `read:packages` scope.
 
-
-
-[Back to top](#toc)
+[Back to top](#table-of-contents)

docs/pull_request_template.md

@@ -39,6 +39,6 @@
 - [ ] My change requires a change to the documentation.
 - [ ] I have updated the documentation accordingly.
 - [ ] I have read the **CONTRIBUTING** document.
-[ODE Contributing Guide](https://github.com/usdot-jpo-ode/jpo-ode/blob/bugfix/Pull_request_template/docs/contributing_guide.md) 
+[ODE Contributing Guide](https://github.com/usdot-jpo-ode/jpo-ode/blob/develop/docs/contributing_guide.md) 
 - [ ] I have added tests to cover my changes.
 - [ ] All new and existing tests passed.

sample.env

@@ -15,7 +15,7 @@
 ####################
 # General Properties
 
-# (Required) The IP address of Docker host machine which can be found by running "ifconfig"
+# (Required) The IP address of Docker host machine which can be found by running "ip -4 addr"
 # Hint: look for "inet addr:" within "eth0" or "en0" for OSX
 DOCKER_HOST_IP=