pre-commit-dbt

List of pre-commit hooks to ensure the quality of your dbt projects.

BETA NOTICE: This tool is still BETA and may have some bugs, so please be forgiving!

Goal

Quick ensure the quality of your dbt projects.

dbt is awesome, but when a number of models, sources, and macros grow it starts to be challenging to maintain quality. People often forget to update columns in schema files, add descriptions, or test. Besides, with the growing number of objects, dbt slows down, users stop running models/tests (because they want to deploy the feature quickly), and the demands on reviews increase.

If this is the case, pre-commit-dbt is here to help you!

List of pre-commit-dbt hooks

💡 Click on hook name to view the details.

Model checks:

• check-column-desc-are-same: Check column descriptions are the same.
• check-model-columns-have-desc: Check the model columns have description.
• check-model-has-all-columns: Check the model has all columns in the properties file.
• check-model-has-description: Check the model has description.
• check-model-has-meta-keys: Check the model has keys in the meta part.
• check-model-has-properties-file: Check the model has properties file.
• check-model-has-tests-by-name: Check the model has a number of tests by test name.
• check-model-has-tests-by-type: Check the model has a number of tests by test type.
• check-model-has-tests: Check the model has a number of tests.
• check-model-tags: Check the model has valid tags.

Script checks:

• check-script-semicolon: Check the script does not contain a semicolon.
• check-script-has-no-table-name: Check the script has not table name (is not using source() or ref() macro for all tables).
• check-script-ref-and-source: Check the script has only existing refs and sources.

Source checks:

• check-source-columns-have-desc: Check for source column descriptions.
• check-source-has-all-columns: Check the source has all columns in the properties file.
• check-source-table-has-description: Check the source table has description.
• check-source-has-freshness: Check the source has the freshness.
• check-source-has-loader: Check the source has loader option.
• check-source-has-meta-keys: Check the source has keys in the meta part.
• check-source-has-tests-by-name: Check the source has a number of tests by test name.
• check-source-has-tests-by-type: Check the source has a number of tests by test type.
• check-source-has-tests: Check the source has a number of tests.
• check-source-tags: Check the source has valid tags.

Modifiers:

• generate-missing-sources: If any source is missing this hook tries to create it.
• generate-model-properties-file: Generate model properties file.
• unify-column-descriptions: Unify column descriptions across all models.
• replace-script-table-names: Replace table names with source() or ref() macros in the script.
• remove-script-semicolon: Remove the semicolon at the end of the script.

dbt commands:

• dbt-clean: Run dbt clean command.
• dbt-compile: Run dbt compile command.
• dbt-deps: Run dbt deps command.
• dbt-docs-generate: Run dbt docs generate command.
• dbt-run: Run dbt run command.
• dbt-test: Run dbt test command.

---

❗If you have an idea for a new hook or you found a bug, let us know❗

Install

For detailed installation and usage, instructions see pre-commit.com site.

pip install pre-commit

Setup

1. Create a file named .pre-commit-config.yaml in your dbt root folder.
2. Add list of hooks you want to run befor every commit. E.g.:

repos:
- repo: https://github.com/offbi/pre-commit-dbt
  rev: v0.1.0
  hooks:
  - id: check-script-semicolon
  - id: check-script-has-no-table-name
  - id: dbt-test
  - id: dbt-docs-generate
  - id: check-model-has-all-columns
    name: Check columns - core
    files: ^models/core
  - id: check-model-has-all-columns
    name: Check columns - mart
    files: ^models/mart
  - id: check-model-columns-have-desc
    files: ^models/mart

3. Optionally, run pre-commit install to set up the git hook scripts. With this, pre-commit will run automatically on git commit! You can also manually run pre-commit run after you stage all files you want to run. Or pre-commit run --all-files to run the hooks against all of the files (not only staged).

Run as Github Action

Unfortunately, you cannot natively use pre-commit-dbt if you are using dbt Cloud. But you can run checks after you push changes into Github.

To do that, make a file .github/workflows/pre-commit.yml.

name: pre-commit

on:
 pull_request:
 push:
 branches: [main]

jobs:
 pre-commit:
 runs-on: ubuntu-latest
 steps:
 - uses: actions/checkout@v2
 - uses: actions/setup-python@v2
 - uses: pre-commit/action@v2.0.0

To run only changed files:

name: pre-commit

on:
 pull_request:
 push:
 branches: [main]

jobs:
 pre-commit:
 runs-on: ubuntu-latest
 steps:
 - uses: actions/checkout@v2
 - uses: actions/setup-python@v2
 - id: file_changes
 uses: trilom/file-changes-action@v1.2.4
 with:
 output: ' '
 - uses: pre-commit/action@v2.0.0
 with:
 extra_args: --files ${{ steps.file_changes.outputs.files}}

To be able to run modifiers you need to use only private repository and change your .github/workflows/pre-commit.yml to:

name: pre-commit

on:
 pull_request:
 push:
 branches: [main]

jobs:
 pre-commit:
 runs-on: ubuntu-latest
 steps:
 - uses: actions/checkout@v2
 with:
 fetch-depth: 0
 - uses: actions/setup-python@v2
 - id: file_changes
 uses: trilom/file-changes-action@v1.2.4
 with:
 output: ' '
 - uses: pre-commit/action@v2.0.0
 with:
 extra_args: --files ${{ steps.file_changes.outputs.files}}
 token: ${{ secrets.GITHUB_TOKEN }}

For more informations about pre-commit/action visit https://github.com/pre-commit/action.