Skip to content
This repository has been archived by the owner on Jul 24, 2024. It is now read-only.

Decouple hook executor from the operator, add Cloud Build execution and adjust testing #48

Open
wants to merge 79 commits into
base: master
Choose a base branch
from
Open
Changes from 1 commit
Commits
Show all changes
79 commits
Select commit Hold shift + click to select a range
0fb5be6
Decouple hook executor from the operator, add Cloud Build execution a…
dinigo Oct 18, 2021
e53b744
Add hook to run in google cloud build
dinigo Oct 18, 2021
b922f21
Fix base operator
dinigo Oct 18, 2021
f2fb018
Test GCP Cloud Build hook
dinigo Oct 18, 2021
ad837ee
Fix cloudbuild logic
dinigo Oct 18, 2021
543d744
Clean unused imports and fix path to upload dbt sources in GCS
dinigo Oct 18, 2021
5ac7f2a
Fix the upload sources test
dinigo Oct 18, 2021
53243c0
Add docstrings to all DbtBaseOperator subclases
dinigo Oct 18, 2021
644f594
Add `env` param to DbtBaseOperator to pass env variables to runtime
dinigo Oct 18, 2021
5cc650d
Export the DbtCloudBuildHook from the hooks package
dinigo Oct 22, 2021
db5e121
Check that google provider API version is the right for the implement…
dinigo Oct 22, 2021
f55624a
Add project_dir flag to the command line
dinigo Oct 22, 2021
ef26f88
Remove cloud build timeout and wait. Thats google provider api v6, not 5
dinigo Oct 22, 2021
129aa93
If dbt_bin is provided None or empty just don't add it to the command
dinigo Oct 22, 2021
289d563
Add `project_dir` option flag
dinigo Oct 22, 2021
081222d
Add the `use_color` flag (#43). If set adds the dbt flag
dinigo Oct 22, 2021
bff2d5c
Template all the fields from the DbtBaseOperator (#46 and #30)
dinigo Oct 22, 2021
6a459e8
Refactor each hook into it's own file
dinigo Oct 25, 2021
dcaa3bb
Remove upload functionality, provided by `LocalFilesystemToGCSOperator`
dinigo Oct 25, 2021
f234900
Adjust imports after refactor
dinigo Oct 25, 2021
c582899
Test Cloud Build DBT hook and remove test cloudbuild upload
dinigo Oct 25, 2021
efdb64d
Fix test params for generate_dbt_cli_command
dinigo Oct 25, 2021
678989c
Export DbtBaseOperator and DbtCloudBuildOperator from the .operators pkg
dinigo Oct 25, 2021
3ff8c4f
Add missing dbt flags, add optional config and Config type to enforce it
dinigo Oct 25, 2021
457f916
Remove dbt_bin from constructors and defer hook instantiation
dinigo Oct 26, 2021
5c3159c
Fix test for TestDbtCliHook checking return code different from 0
dinigo Oct 26, 2021
bec7fe5
Bring back "dir" to fix back the API, add warnings
dinigo Oct 26, 2021
0c7587a
DbtCleanOperator was missing from operator exports (__init__)
dinigo Oct 26, 2021
ce732b2
Fix typo (thanks @fenimore) and reorganize imports
dinigo Oct 26, 2021
8ea9958
Comply with linter (flake8)
dinigo Oct 26, 2021
f1c84fe
Refactor, move google operator out of the dbt_operator
dinigo Oct 27, 2021
a3e1213
Display Cloud Build logs in the Airflow logs
dinigo Oct 27, 2021
7776c45
Get rid of __init__ files exporting classes from different places
dinigo Oct 27, 2021
2f8b9e7
Fix TestDbtCloudBuildHook with the new config sent to Cloud Build
dinigo Oct 27, 2021
3c90bc5
Remove unused DbtCloudBuildHook param `dbt_bin`
dinigo Oct 27, 2021
abd4bfe
Test extensively command generation, execution in cli, hook injection...
dinigo Nov 2, 2021
13b2559
Replace explicit generate_dbt_cli_command with generic one
dinigo Nov 2, 2021
0435b58
Docs adjustments mainly
dinigo Nov 2, 2021
e3736f5
Flags are passed True to be reflected in the command line.
dinigo Nov 2, 2021
82ec4f4
Make abstract method implement a single LOC `pass`. thanks @fenimore
dinigo Dec 7, 2021
b505d59
Explain conditional import of TypedDict as suggested by @andrewrjones
dinigo Dec 7, 2021
239e05f
Reuse type in different conditionals, suggested by @johanna-ojeling-d…
dinigo Dec 7, 2021
0cc0205
Make project_id optional in the DbtCloudBuildHook
dinigo Dec 7, 2021
a1dbbc7
Fix typo in pydoc
dinigo Dec 7, 2021
cb0566d
Make `env` dict for environment variables for google dbt hook optional
dinigo Dec 7, 2021
d4e5184
Make `service_account ` for google dbt hook optional
dinigo Dec 7, 2021
98c90f0
Shorten up conditional assignment of project_id
dinigo Dec 7, 2021
33af351
Make `env` dict for environment variables for base dbt hook optional
dinigo Dec 7, 2021
139bfe3
Shorten up conditional assignment of env dict
dinigo Dec 7, 2021
c6b3699
Make `env ` for base dbt hook optional
dinigo Dec 7, 2021
face550
Add descriptions to each parametrized dbt test
dinigo Dec 7, 2021
6f7a507
Add README.md code examples
dinigo Dec 7, 2021
1c41b36
Split check values in two for readability
dinigo Dec 7, 2021
52884ae
Extract provider version check to a function and test it, also typos
dinigo Dec 7, 2021
df62625
Fix conditionals comparing with None since None is `falsy` in python
dinigo Dec 7, 2021
642f6dd
Fix python files not being detected in setup by `find_packages`:
dinigo Jan 13, 2022
614309b
Bump version
dinigo Jan 13, 2022
23e16a6
Make operator font color white, easier to read against a mid-dark bg
dinigo Jan 13, 2022
ce219a0
Allow false flags and reorder dbt render
dinigo Jan 14, 2022
22ddab1
Share DbtBaseOperator props between subclasses
dinigo Jan 14, 2022
de1fec4
Ignore deprecated @apply_defaults, It's needed for backwards compat
dinigo Jan 14, 2022
0f6e313
Fix dbt config, some params for dbt and some for the specific command
dinigo Jan 14, 2022
f2075f5
Allow a custom image other than the official from fishtown
dinigo Jan 14, 2022
b279112
Add DbtCloudBuildOperator templated fields to the DbtBaseOperator ones
dinigo Jan 14, 2022
957f76b
Bump dbt docker image for cloud build
dinigo Jan 14, 2022
a2a3346
Cloud build hook can raise silently. Surround with a try/catch
dinigo Jan 14, 2022
4515199
Add default conn_id, dbt_version and dbt_image and simpler cloudbuild…
dinigo Jan 14, 2022
f42cdfe
Fix compoundd commands like 'dbt docs generate' not running in GCP
dinigo Feb 1, 2022
0c1bcf4
Export artifacts (`target` folder) when running in cloud build:
dinigo Feb 1, 2022
697a2ba
Enhance type safety
dinigo Feb 1, 2022
460d361
Use `Dict` instead of the default new `dict` for backwards compat
dinigo Mar 8, 2022
7cab0fd
use the cloud build library instead of the hook
dinigo Mar 8, 2022
b3dbdb6
Ignore terraform, dbt and pycharm related files
dinigo Mar 8, 2022
e1969f8
Bump version
dinigo Mar 8, 2022
9766bbc
chore: update default dbt-imge to github registry
dinigo Oct 24, 2022
f1744a8
feat: add debug operator
dinigo Oct 24, 2022
f34d22e
chore: update default dbt docker image versions, part 2
dinigo Oct 24, 2022
09548af
chore: simpler and standard __version__ so that compilation picks it
dinigo Oct 24, 2022
98692ed
ci: replace old build system with the new default pyproject
dinigo Oct 24, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
chore: simpler and standard __version__ so that compilation picks it
dinigo committed Oct 24, 2022

Verified

This commit was signed with the committer’s verified signature.
dinigo Daniel Iñigo
commit 09548af8bfd8e70356be629a756a828a0bdb3e30
4 changes: 1 addition & 3 deletions airflow_dbt/__version__.py
Original file line number Diff line number Diff line change
@@ -1,3 +1 @@
VERSION = (0, 5, 7)

__version__ = '.'.join(map(str, VERSION))
__version__ = "0.5.10"