From ee5cf328f7af91222c423921c4d81fc8d33c9794 Mon Sep 17 00:00:00 2001 From: Jennifer Power Date: Wed, 3 Jan 2024 08:59:41 -0500 Subject: [PATCH] chore: docs and config maintenance (#105) * chore: update semgrep version in pre-commit and add linting check Bumps semgrep version to 1.51.0 and adds a local check for `make lint` Signed-off-by: Jennifer Power * chore: removes warning linters from megalinter configs To ensure warnings/violations are not missed, remove the linter with disable errors and move to enabled. Signed-off-by: Jennifer Power * chore: adds pre-commit setup in Dev Container configuration Signed-off-by: Jennifer Power * docs: updates contributing guide with information about development tools Signed-off-by: Jennifer Power * docs: adds a table of contents to CONTRIBUTING.md Signed-off-by: Jennifer Power --------- Signed-off-by: Jennifer Power --- .devcontainer/devcontainer.json | 2 +- .mega-linter.yml | 4 -- .pre-commit-config.yaml | 12 +++++- CONTRIBUTING.md | 73 ++++++++++++++++++++++++++++----- Makefile | 4 +- 5 files changed, 75 insertions(+), 20 deletions(-) diff --git a/.devcontainer/devcontainer.json b/.devcontainer/devcontainer.json index a140768f..fb0eaf4c 100644 --- a/.devcontainer/devcontainer.json +++ b/.devcontainer/devcontainer.json @@ -29,7 +29,7 @@ "POETRY_VIRTUALENVS_IN_PROJECT": "true" }, "updateContentCommand": "python3 -m venv $VENV_PATH && source $VENV_PATH/bin/activate && poetry install --no-root --no-interaction", - "postCreateCommand": "echo \"source $VENV_PATH/bin/activate\" >> $HOME/.bashrc", + "postCreateCommand": "echo \"source $VENV_PATH/bin/activate\" >> $HOME/.bashrc && make pre-commit", "workspaceMount": "source=${localWorkspaceFolder},target=/workspace,type=bind", "workspaceFolder": "/workspace" } \ No newline at end of file diff --git a/.mega-linter.yml b/.mega-linter.yml index af5e5649..4a414fec 100644 --- a/.mega-linter.yml +++ b/.mega-linter.yml @@ -4,10 +4,6 @@ ENABLE_LINTERS: - REPOSITORY_KICS - ACTION_ACTIONLINT - MARKDOWN_MARKDOWNLINT - - MAKEFILE_CHECKMAKE - -DISABLE_ERRORS_LINTERS: - - MAKEFILE_CHECKMAKE - BASH_SHELLCHECK REPOSITORY_KICS_ARGUMENTS: "--fail-on high" \ No newline at end of file diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index fdaa4148..4590e25a 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -1,6 +1,6 @@ repos: - repo: https://github.com/returntocorp/semgrep - rev: v1.42.0 + rev: v1.51.0 hooks: - id: semgrep name: Semgrep Python @@ -10,4 +10,12 @@ repos: name: Semgrep Bandit types: [python] exclude: "^tests/.+$" - args: ["--config", "p/bandit", "--error", --metrics=off] \ No newline at end of file + args: ["--config", "p/bandit", "--error", --metrics=off] + - repo: local + hooks: + - id: lint + language: script + name: Check formatting and lint + entry: make lint + stages: [commit] + types: [python] \ No newline at end of file diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index dd022e03..50a608dc 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -6,7 +6,22 @@ Before you start contributing, please take a moment to read through the guide be > WARNING: This project is currently under initial development. APIs may be changed incompatibly from one commit to another. -### How To Contribute +- [Contributing to trestlebot](#contributing-to-trestlebot) + - [How To Contribute](#how-to-contribute) + - [Opening a Pull Request](#opening-a-pull-request) + - [Developer Guide](#developer-guide) + - [Prerequisites](#prerequisites) + - [How It Works](#how-it-works) + - [Components](#components) + - [Code structure](#code-structure) + - [Format and Styling](#format-and-styling) + - [Update the `actions` files](#update-the-actions-files) + - [Running tests](#running-tests) + - [Run with poetry](#run-with-poetry) + - [Local testing](#local-testing) + + +## How To Contribute Some initial contributions could be: @@ -29,35 +44,71 @@ When submitting a pull request, please follow these guidelines: - [Python](https://www.python.org/downloads/) - v3.8+ - [Poetry](https://python-poetry.org/) -- [Podman](https://podman.io/docs/installation) +- [Podman](https://podman.io/docs/installation) (Optional) - For testing locally and end-to-end tests -### How It Works +## Development Environment + +For a reproducible development environment, we use Dev Containers. See [devcontainer.json](./.devcontainer/devcontainer.json) for more information. Note that this does not include the `podman` installation to avoid the requirement for containers with elevated privileges. + +## How It Works For workflow diagrams, see the [diagrams](./docs/diagrams/) under the `docs` folder. -#### Components +### Components 1. CI Provider - Runs or builds and runs trestle-bot container 2. Trestle Bot - Provides logic for managing workspace and containerized environment for use in workflows 3. Compliance-Trestle - Upstream library that provides core logic for how OSCAL content is managed -#### Code structure +### Code structure - `actions` - Provides specific logic for `trestle-bot` tasks that are packaged as Actions. See [README.md](./actions/README.md) for more information. - `entrypoints` - Provides top level logic for specific user-facing tasks. These tasks are not necessarily related in any way so they are not organized into a hierarchical command structure, but they do inherit logic and flags from a base class. - `provider.py, github.py, and gitlab.py` - Git provider abstract class and concrete implementations for interacting with the API. - `tasks` - Pre-tasks can be configured before the main git logic is run. Any task that does workspace management should go here. - `tasks/authored` - The `authored` package contains logic for managing authoring tasks for single instances of a top-level OSCAL model. These encapsulate logic from the `compliance-trestle` library and allows loose coupling between `tasks` and `authored` types. -- `transformers` - This contains data transformation logic; specifically for rules. +- `transformers` - This contains data transformation logic; specifically for rules. + -### Format and Styling +## Format and Styling + +This project uses `black` and `isort` for formatting and `flake8` for linting. You can run these commands to format and lint your code. +They are also run as part as a pre-commit hook. ```bash make format -make lint +make lint- [Contributing to trestlebot](#contributing-to-trestlebot) + - [How To Contribute](#how-to-contribute) + - [Opening a Pull Request](#opening-a-pull-request) + - [Developer Guide](#developer-guide) + - [Prerequisites](#prerequisites) + - [How It Works](#how-it-works) + - [Components](#components) + - [Code structure](#code-structure) + - [Format and Styling](#format-and-styling) + - [Update the `actions` files](#update-the-actions-files) + - [Running tests](#running-tests) + - [Run with poetry](#run-with-poetry) + - [Local testing](#local-testing) + ``` -### Running tests +For non-Python files, we use [Megalinter](https://github.com/oxsecurity/megalinter) to lint in a CI task. See [megalinter.yaml](./.github/megalinter.yaml) for more information. + +## Type Hints and Static Type Checking + +We encourage the use of type hints in Python code to enhance readability, maintainability, and robustness of the codebase. Type hints serve as documentation and aid in catching potential errors during development. For static type analysis, we utilize `mypy`. Running `make lint` will run `mypy` checks on the codebase. + +## Analysis Tools + +- [SonarCloud](https://sonarcloud.io/dashboard?id=rh-psce_trestle-bot) - We use SonarCloud to analyze code quality, coverage, and security. To not break GitHub security model, this will not run on a forked repository. +- [Semgrep](https://semgrep.dev/docs/extensions/overview/#pre-commit) - Identify issues in the local development environment before committing code. These checks are also run in CI. + +## Running tests + +Run all tests with `make test` or `make test-slow` to run all tests including end-to-end. +For information on end-to-end tests, see [README.md](./tests/e2e/README.md). + ```bash # Run all tests make test @@ -69,10 +120,10 @@ make test-e2e ### Run with poetry ``` -poetry shell -poetry install +make develop poetry run trestlebot-autosync poetry run trestlebot-rules-transform +poetry run trestlebot-create-cd ``` ### Local testing diff --git a/Makefile b/Makefile index 4f908e99..d2dd5d5e 100644 --- a/Makefile +++ b/Makefile @@ -6,7 +6,7 @@ all: develop lint test .PHONY: all develop: pre-commit - @poetry install + @poetry install @poetry shell .PHONY: develop @@ -44,7 +44,7 @@ test-code-cov: .PHONY: test-code-cov # https://github.com/python-poetry/poetry/issues/994#issuecomment-831598242 -# Check for CVEs locally. For automated/scheduled checks, use dependabot. +# Check for CVEs locally. For continuous dependency updates, we use dependabot. dep-cve-check: @poetry export -f requirements.txt --without-hashes | poetry run safety check --stdin .PHONY: dep-cve-check