Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feature/performance-enhancement #39

Merged
merged 19 commits into from
Apr 30, 2024

Conversation

fivetran-catfritz
Copy link
Contributor

@fivetran-catfritz fivetran-catfritz commented Apr 4, 2024

PR Overview

This PR will address the following Issue/Feature:

  • internal ticket

This PR will result in the following new package version:

  • v0.7.0: breaking due to materialization changes requiring full refresh

Please provide the finalized CHANGELOG entry which details the relevant changes included in this PR:

🚨 Breaking Changes 🚨

  • To reduce storage, updated default materialization of staging models to views.

⚠️ Running a --full-refresh will be required if you have previously run these staging models as tables and get the following error:

Trying to create view <model path> but it currently exists as a table. Either drop <model path> manually, or run dbt with `--full-refresh` and dbt will drop it for you.

Under the Hood:

  • Included auto-releaser GitHub Actions workflow to automate future releases.
  • Incorporated the new fivetran_utils.drop_schemas_automation macro into the end of each Buildkite integration test job.
  • Updated the pull request templates.

PR Checklist

Basic Validation

Please acknowledge that you have successfully performed the following commands locally:

  • dbt run –full-refresh && dbt test
  • dbt run (if incremental models are present) && dbt test

Before marking this PR as "ready for review" the following have been applied:

  • The appropriate issue has been linked, tagged, and properly assigned.
  • All necessary documentation and version upgrades have been applied.
  • docs were regenerated (unless this PR does not include any code or yml updates).
  • BuildKite integration tests are passing.
  • Detailed validation steps have been provided below.

Detailed Validation

Please share any and all of your validation steps:

  • These were only default materialization setting changes and under the hood changes.

If you had to summarize this PR in an emoji, which would it be?

💃

@fivetran-catfritz fivetran-catfritz self-assigned this Apr 8, 2024
Copy link
Contributor

@fivetran-joemarkiewicz fivetran-joemarkiewicz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@fivetran-catfritz this is looking great, I just have a few quick callouts that I want your thoughts on before approving.

.buildkite/scripts/run_models.sh Outdated Show resolved Hide resolved
.buildkite/scripts/run_models.sh Outdated Show resolved Hide resolved
integration_tests/dbt_project.yml Outdated Show resolved Hide resolved
integration_tests/requirements.txt Outdated Show resolved Hide resolved
macros/is_databricks_sql_warehouse.sql Outdated Show resolved Hide resolved
Copy link
Contributor Author

@fivetran-catfritz fivetran-catfritz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @fivetran-joemarkiewicz! Updated based on your comments, and this is ready for re-review.

.buildkite/scripts/run_models.sh Outdated Show resolved Hide resolved
.buildkite/scripts/run_models.sh Outdated Show resolved Hide resolved
integration_tests/dbt_project.yml Outdated Show resolved Hide resolved
macros/is_databricks_sql_warehouse.sql Outdated Show resolved Hide resolved
integration_tests/requirements.txt Outdated Show resolved Hide resolved
Copy link
Contributor

@fivetran-joemarkiewicz fivetran-joemarkiewicz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@fivetran-catfritz fivetran-catfritz merged commit 11307b2 into main Apr 30, 2024
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants