feature/reduce-date-spine

[fivetran-catfritz]

PR Overview

This PR will address the following Issue/Feature:

• Internal ticket

Submission Checklist

Submitter:

• Alignment meeting with the reviewer
• Provide validation details:
  • Validation Steps: Outline how to verify the changes
  • Testing Instructions: Clear steps for running/testing (e.g., scripts, sample data)
  • Focus Areas: Highlight any complex logic or queries needing special attention
• Changelog outline

Reviewer:

• Confirm submission requirements are met

Changelog

• Full draft after PR approval

• Validation:

  • Hex to show the incremental works over a couple days, 1 insert_overwrite & 1 delete+insert
    • See internal ticket
  • Confirm consistency tests pass
    • See internal ticket for vars
    • The consistency test passes in both production and development since the date spine output remains unchanged. The update uses less resources to generate the base date spine that the downstream CTEs operate on.
      • 
  • Is runtime better on Snowflake
    • When running in a test environment, the execution times do improve ( vs
      ), however the customer test (if we can get one) will be the most valuable.

[fivetran-joemarkiewicz]

@fivetran-catfritz thanks for applying these updates and for the detailed validation steps. This PR looks good to me and ready for release review!

• ✅ Confirmed all deleted variables from the dbt_project.yml either are no longer used or have a default value provided each time they are referenced in the code.
• ✅ Confirmed all mixpanel.date_today() macro references have been updated to use current_date.
• ✅ Confirmed the dbt compile command works on first run to validate the flags.WHICH update to the date spine model.
• ✅ Confirmed all validations have been provided and are passing or show expected results.
• ✅ Confirmed for each incremental end model appropriately included new records on incremental runs.
• ✅ Confirmed that subsequent incremental runs are idempotent

[fivetran-joemarkiewicz]

I'd also recommend adding an under the hood piece detailing why we're removing the variable declarations in the dbt_project.yml. Mainly for our reference in the future, but also good to callout since that may look like a big change, when it's really not.

[fivetran-catfritz]

Updated!

[fivetran-joemarkiewicz]

Looks like it's really appropriate to remove this since we essentially deprecated a few releases ago.

[fivetran-reneeli]

Thanks @fivetran-catfritz just had a question!

[fivetran-reneeli]

Should the default date '2010-01-01' in line 28 also be removed?

[fivetran-catfritz]

I considered it, but it should stay. The issue with the date spine was that it internally generates all dates back to 2010 even if there weren't records that went back that far, whereas this model only references existing records. We need a default value of some sort but I wanted to avoid pinging the source and incurring costs if we didn't need to.

[fivetran-reneeli]

Gotcha! Thanks for explaining!

[fivetran-reneeli]

LGTM