Skip to content

chore: docs and config maintenance #105

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Jan 3, 2024
Merged

chore: docs and config maintenance #105

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
ca65048
chore: update semgrep version in pre-commit and add linting check
jpower432 Dec 7, 2023
b5b4343
chore: removes warning linters from megalinter configs
jpower432 Dec 7, 2023
b52d864
chore: adds pre-commit setup in Dev Container configuration
jpower432 Dec 7, 2023
daaf9f6
docs: updates contributing guide with information about development t…
jpower432 Dec 7, 2023
890886f
docs: adds a table of contents to CONTRIBUTING.md
jpower432 Jan 2, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .devcontainer/devcontainer.json
View file
Open in desktop
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wanted to understand why you added the && make pre-commit for my own benefit, we can discuss during our next daily

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Understood. This was added to make sure that the pre-commit hooks are installed in the dev container environment. They can be uninstalled later if the dev using the environment wants to do that, but it makes sense to me to install them by default to reduce errors and additional steps for the environment configuration.

Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@
"POETRY_VIRTUALENVS_IN_PROJECT": "true"
},
"updateContentCommand": "python3 -m venv $VENV_PATH && source $VENV_PATH/bin/activate && poetry install --no-root --no-interaction",
"postCreateCommand": "echo \"source $VENV_PATH/bin/activate\" >> $HOME/.bashrc",
"postCreateCommand": "echo \"source $VENV_PATH/bin/activate\" >> $HOME/.bashrc && make pre-commit",
"workspaceMount": "source=${localWorkspaceFolder},target=/workspace,type=bind",
"workspaceFolder": "/workspace"
}
4 changes: 0 additions & 4 deletions .mega-linter.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,10 +4,6 @@ ENABLE_LINTERS:
- REPOSITORY_KICS
- ACTION_ACTIONLINT
- MARKDOWN_MARKDOWNLINT
- MAKEFILE_CHECKMAKE

DISABLE_ERRORS_LINTERS:
- MAKEFILE_CHECKMAKE
- BASH_SHELLCHECK

REPOSITORY_KICS_ARGUMENTS: "--fail-on high"
12 changes: 10 additions & 2 deletions .pre-commit-config.yaml
View file
Open in desktop
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wanted to understand the hooks added here

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Linting checks are already run in CI for verification, but it is faster to have them run as a pre-commit check. Currently make lint is just run manually, but adding will reduce the chance that the pipeline will fail only because of a linting error. This is running the same command as the check in CI to ensure that the version of the linters are the same and run with the same configuration.

Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
repos:
- repo: https://github.com/returntocorp/semgrep
rev: v1.42.0
rev: v1.51.0
hooks:
- id: semgrep
name: Semgrep Python
Expand All @@ -10,4 +10,12 @@ repos:
name: Semgrep Bandit
types: [python]
exclude: "^tests/.+$"
args: ["--config", "p/bandit", "--error", --metrics=off]
args: ["--config", "p/bandit", "--error", --metrics=off]
- repo: local
hooks:
- id: lint
language: script
name: Check formatting and lint
entry: make lint
stages: [commit]
types: [python]
73 changes: 62 additions & 11 deletions CONTRIBUTING.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,22 @@ Before you start contributing, please take a moment to read through the guide be

> WARNING: This project is currently under initial development. APIs may be changed incompatibly from one commit to another.

### How To Contribute
- [Contributing to trestlebot](#contributing-to-trestlebot)
- [How To Contribute](#how-to-contribute)
- [Opening a Pull Request](#opening-a-pull-request)
- [Developer Guide](#developer-guide)
- [Prerequisites](#prerequisites)
- [How It Works](#how-it-works)
- [Components](#components)
- [Code structure](#code-structure)
- [Format and Styling](#format-and-styling)
- [Update the `actions` files](#update-the-actions-files)
- [Running tests](#running-tests)
- [Run with poetry](#run-with-poetry)
- [Local testing](#local-testing)


## How To Contribute

Some initial contributions could be:

Expand All @@ -29,35 +44,71 @@ When submitting a pull request, please follow these guidelines:

- [Python](https://www.python.org/downloads/) - v3.8+
- [Poetry](https://python-poetry.org/)
- [Podman](https://podman.io/docs/installation)
- [Podman](https://podman.io/docs/installation) (Optional) - For testing locally and end-to-end tests

### How It Works
## Development Environment

For a reproducible development environment, we use Dev Containers. See [devcontainer.json](./.devcontainer/devcontainer.json) for more information. Note that this does not include the `podman` installation to avoid the requirement for containers with elevated privileges.

## How It Works

For workflow diagrams, see the [diagrams](./docs/diagrams/) under the `docs` folder.

#### Components
### Components

1. CI Provider - Runs or builds and runs trestle-bot container
2. Trestle Bot - Provides logic for managing workspace and containerized environment for use in workflows
3. Compliance-Trestle - Upstream library that provides core logic for how OSCAL content is managed

#### Code structure
### Code structure

- `actions` - Provides specific logic for `trestle-bot` tasks that are packaged as Actions. See [README.md](./actions/README.md) for more information.
- `entrypoints` - Provides top level logic for specific user-facing tasks. These tasks are not necessarily related in any way so they are not organized into a hierarchical command structure, but they do inherit logic and flags from a base class.
- `provider.py, github.py, and gitlab.py` - Git provider abstract class and concrete implementations for interacting with the API.
- `tasks` - Pre-tasks can be configured before the main git logic is run. Any task that does workspace management should go here.
- `tasks/authored` - The `authored` package contains logic for managing authoring tasks for single instances of a top-level OSCAL model. These encapsulate logic from the `compliance-trestle` library and allows loose coupling between `tasks` and `authored` types.
- `transformers` - This contains data transformation logic; specifically for rules.
- `transformers` - This contains data transformation logic; specifically for rules.


### Format and Styling
## Format and Styling

This project uses `black` and `isort` for formatting and `flake8` for linting. You can run these commands to format and lint your code.
They are also run as part as a pre-commit hook.

```bash
make format
make lint
make lint- [Contributing to trestlebot](#contributing-to-trestlebot)
- [How To Contribute](#how-to-contribute)
- [Opening a Pull Request](#opening-a-pull-request)
- [Developer Guide](#developer-guide)
- [Prerequisites](#prerequisites)
- [How It Works](#how-it-works)
- [Components](#components)
- [Code structure](#code-structure)
- [Format and Styling](#format-and-styling)
- [Update the `actions` files](#update-the-actions-files)
- [Running tests](#running-tests)
- [Run with poetry](#run-with-poetry)
- [Local testing](#local-testing)

```

### Running tests
For non-Python files, we use [Megalinter](https://github.com/oxsecurity/megalinter) to lint in a CI task. See [megalinter.yaml](./.github/megalinter.yaml) for more information.

## Type Hints and Static Type Checking

We encourage the use of type hints in Python code to enhance readability, maintainability, and robustness of the codebase. Type hints serve as documentation and aid in catching potential errors during development. For static type analysis, we utilize `mypy`. Running `make lint` will run `mypy` checks on the codebase.

## Analysis Tools

- [SonarCloud](https://sonarcloud.io/dashboard?id=rh-psce_trestle-bot) - We use SonarCloud to analyze code quality, coverage, and security. To not break GitHub security model, this will not run on a forked repository.
- [Semgrep](https://semgrep.dev/docs/extensions/overview/#pre-commit) - Identify issues in the local development environment before committing code. These checks are also run in CI.

## Running tests

Run all tests with `make test` or `make test-slow` to run all tests including end-to-end.
For information on end-to-end tests, see [README.md](./tests/e2e/README.md).

```bash
# Run all tests
make test
Expand All @@ -69,10 +120,10 @@ make test-e2e

### Run with poetry
```
poetry shell
poetry install
make develop
poetry run trestlebot-autosync
poetry run trestlebot-rules-transform
poetry run trestlebot-create-cd
```

### Local testing
Expand Down
4 changes: 2 additions & 2 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ all: develop lint test
.PHONY: all

develop: pre-commit
@poetry install
@poetry install
@poetry shell
.PHONY: develop

Expand Down Expand Up @@ -44,7 +44,7 @@ test-code-cov:
.PHONY: test-code-cov

# https://github.com/python-poetry/poetry/issues/994#issuecomment-831598242
# Check for CVEs locally. For automated/scheduled checks, use dependabot.
# Check for CVEs locally. For continuous dependency updates, we use dependabot.
dep-cve-check:
@poetry export -f requirements.txt --without-hashes | poetry run safety check --stdin
.PHONY: dep-cve-check
Expand Down