3.0.2

What's Changed

• Fix catchall errors by @Ekrekr in #1802

Full Changelog: 3.0.1...3.0.2

3.0.1

Important: Prefer 3.0.2 over this release - this release has issues.

See #1800 for more context.

What's Changed

• Enable File-Specific Formatting in dataform format by @hiracky16 in #1754
• Assertions configs as protos by @Ekrekr in #1731
• Fix vscode extension crashes on windows by @moker-spaghetti in #1751
• Update the version of the vscode extension by @GJMcGowan in #1759
• Fix empty object internal error by @Ekrekr in #1763
• Split table tests by @Ekrekr in #1761
• Fix protected default, add a unit test for SQLX config parameters by @Ekrekr in #1718
• Declaration configs as protos by @Ekrekr in #1767
• Add support for action configs table assertions and hermeticity, add action configs fields tests by @Ekrekr in #1762
• Rethrow useful errors when reading workflow_settings.yaml by @Ekrekr in #1772
• View config as protos, via a new class by @Ekrekr in #1760
• Incremental tables as protos, via a new class by @Ekrekr in #1773
• Operation config as protos by @Ekrekr in #1766
• Fix vscode ref issue #1768 by @moker-spaghetti in #1769
• Add workflow_settings.yaml validation/completion VSCode extension by @moker-spaghetti in #1770
• fix(1776): dry run of assertions failing by @ashish10alex in #1778
• feat: vscode extension, more intutive compile errors by @ashish10alex in #1775
• Update vscode version to incorporate recent changes by @GJMcGowan in #1784
• Added proto definitions for Data Preparations by @fernst in #1788
• Handling for Data Preparation actions by @fernst in #1789
• Support building on arm Mac, Linux environments by @bmagyarkuti in #1790
• fix(1781): go to definition when namePrefix is specified in workflow_settings.yaml by @ashish10alex in #1783
• Fix vscode server linting by @Ekrekr in #1797
• Make the stateless CLI package installation error clearer by @Ekrekr in #1795
• Regenerate configs reference with camelCase fields by @Ekrekr in #1796
• Fix notebook dependency targets by @Ekrekr in #1792
• Bumped version to 3.0.1 by @fernst in #1798
• Remove some outdated configs references, fix generated formatting by @Ekrekr in #1799

New Contributors

• @hiracky16 made their first contribution in #1754
• @moker-spaghetti made their first contribution in #1751
• @ashish10alex made their first contribution in #1778
• @fernst made their first contribution in #1788

Full Changelog: 3.0.0...3.0.1

3.0.0

TL;DR of What's Changed Since 2.9.0

dataform.json -> workflow_settings.yaml

workflow_settings.yaml has been introduced, which will gradually replace dataform.json in a later version; there is no immediate action to be taken, as dataform.json files are still valid in projects with Dataform Core 3.0.0.

dataform.json is being deprecated in favor of workflow_settings.yaml. This means that:

• Workflow settings are now strictly typed, in Protobuf format.
• The Dataform Core version can be specified directly in the workflow_settings.yaml file. Note: to have more than just @dataform/core as a dependency, a package.json must still be used.

Example conversion of workflow_settings.yaml:

defaultProject: dataform-demos
defaultLocation: us
defaultDataset: dataform
defaultAssertionDataset: dataform_assertions
dataformCoreVersion: 3.0.0
vars:
    environmentName: "development"

The above is equivalent to the dataform.json file:

{
  "warehouse": "bigquery",
  "defaultDatabase": "dataform-demos",
  "defaultLocation": "us",
  "defaultSchema": "dataform",
  "assertionSchema": "dataform_assertions"
  "vars": {
    "environmentName": "development"
  }
}

Notebooks Actions and actions.yaml

Notebooks as Dataform actions are on their way - but not quite yet! They're part of the compiled graph, and soon they'll be executable.

A new way of configuring action configs through actions.yaml has been implemented to support this.

An example of loading a notebook in Dataform can be seen at https://github.com/dataform-co/dataform/tree/main/examples/extreme_weather_programming.

Stateless Package Installation by @dataform/cli

Package installation by @dataform/cli is now stateless! The CLI will install NPM packages during compilation if version is defined in the workflow_settings.yaml file.

This means no node_modules folder has to be seen in the project, and Dataform users no longer need to be familiar with NPM.

Compilation Output is Now Warehouse Agnostic

Previously the output of compilation results from @dataform/core would insert warehouse specific SQL into the compiled graph. Where possible, this has been removed - transferring the responsibility of inserting warehouse specific SQL into whichever execution engine is running Dataform.

Additionally, support for non-BigQuery warehouses has been dropped. We're in discussions with Datashell for them to provide a warehouse-agnostic CLI execution engine based off of Dataform compiled graphs. In the meantime however, if you need support for a non-BigQuery warehouse, please continue using the latest version starting with 2.x.x!

dependOnDependencyAssertions

An easier ways to add assertions from dependency as dependencies has been introduced.

dependOnDependencyAssertions in config blocks can be used to add assertions from all dependencies of the action as dependencies.

config { 
    type: "view",
    dependOnDependencyAssertions: true,
    dependencies: ["some_table"]
}

select test from ${ref("some_other_table")}

Additionally, the includeDependentAssertions parameter can be used when setting individual dependencies either in config.dependencies or in ref() to add assertions for these dependencies as the dependencies for current action.

config { 
    type: "view",
    dependencies: [{name: "some_table", includeDependentAssertions: true}]
}

select test from ${ref({name: "some_other_table", includeDependentAssertions: true})}

Full Changelog from 2.9.0: 2.9.0...3.0.0

3.0.0-beta.6

What's Changed

• Fix and re-enable integration tests by @Ekrekr in #1729
• Make example tests run off of the CLI, tidy testing/ by @Ekrekr in #1730
• Bump semver from 5.6.0 to 7.6.0 by @dependabot in #1724
• Some project config improvements by @Ekrekr in #1732
• Make proto object verification give more descriptive error messages by @Ekrekr in #1733
• Add tests for locking in SQLX config options validity by @Ekrekr in #1734
• Add tests for compilation errors where they're missing by @Ekrekr in #1735
• Fix bigquery dryrun by @Ekrekr in #1742
• Make formatting in the CLI use the same SQL formatter as BigQuery by @Ekrekr in #1741

Full Changelog: 3.0.0-beta.5...3.0.0-beta.6

3.0.0-beta.5

The description of dependOnDependencyAssertions has been moved to https://github.com/dataform-co/dataform/releases/tag/3.0.0.

What's Changed

• Update to using GitHub actions for tests by @Ekrekr in #1708
• Re-add cloud build while a new GCP project is set up for GitHub actions by @Ekrekr in #1710
• Update contributing tests guide to just test core by @Ekrekr in #1711
• Bump tar from 6.1.11 to 6.2.1 by @dependabot in #1715
• Bump semver from 5.6.0 to 7.6.0 by @dependabot in #1716
• Bump protobufjs from 7.2.4 to 7.2.5 by @dependabot in #1714
• Update cloudbuild file to default name, make yaml by @Ekrekr in #1722
• Make cloud build for dataform-open-source work for pull requests by @Ekrekr in #1728
• Feature: dependent assertions as dependencies by @Tuseeq1 in #1720

New Contributors

• @Tuseeq1 made their first contribution in #1720

Full Changelog: 3.0.0-beta.4...3.0.0-beta.5

3.0.0-beta4

What's Changed

• Expose defaultLocation in dataform.projectConfig. by @BenBirt in #1705
• Publish 3.0.0-beta.4. by @BenBirt in #1706

Full Changelog: 3.0.0-beta.3...3.0.0-beta.4

3.0.0-beta.3

What's Changed

• Add notebook runtime options to workflow settings by @Ekrekr in #1686
• Add a configs reference page, and tidy up other readmes by @Ekrekr in #1688
• Bump follow-redirects from 1.15.4 to 1.15.6 by @dependabot in #1690
• Remove rules docker by @Ekrekr in #1689
• Fix credentials path resolution when projectDir is specified by @Ekrekr in #1694
• Fix CLI tags, allow comma delisted tags, add test by @Ekrekr in #1695
• Prevent confusion around JSON printing of the execution graph by @Ekrekr in #1697
• Extend dryrun to do dry runs against BigQuery by @Ekrekr in #1698
• Add bool argument to disable, default true to keep the API the same by @Ekrekr in #1700
• Throw errors when dots are in fields in targets by @Ekrekr in #1699
• Normalize paths during compilation by @Ekrekr in #1702

Full Changelog: 3.0.0-beta.1...3.0.0-beta.3

3.0.0-beta1

What's Changed

• Remove all analytics from the dataform CLI by @GJMcGowan in #1683
• Remove a bunch of random packages we don't use by @GJMcGowan in #1684
• Add notebook options to configs proto by @Ekrekr in #1685

Full Changelog: 3.0.0-beta.0...3.0.0-beta.1

3.0.0-beta.0

Description of changes since 2.9.0 have been moved to https://github.com/dataform-co/dataform/releases/tag/3.0.0.

What's Changed

• Remove non bigquery warehouses by @Ekrekr in #1550
• Remove dynamic warehouse inference, and more clearly differentiate core and api's SQL adapters by @Ekrekr in #1554
• Consolidate cli package by @Ekrekr in #1557
• Merge main to v3 by @Ekrekr in #1563
• Deprecate support for old type .sql files by @Ekrekr in #1564
• Add initial action config proto definitions by @Ekrekr in #1576
• Merge main to main_v3 by @Ekrekr in #1579
• Remove gen index in favor of main by @Ekrekr in #1571
• Read workflow_settings.yaml by @Ekrekr in #1580
• Validate workflow settings fields by @Ekrekr in #1581
• Remove deprecated run cache and gcloud project ID fields by @Ekrekr in #1584
• Move concurrency and action retries from project config proto to the CLI by @Ekrekr in #1585
• Merge main to main_v3 3 by @Ekrekr in #1590
• Populate warehouse as BigQuery by default by @Ekrekr in #1591
• Update v3 to use the apache 2.0 license by @Ekrekr in #1594
• Stop using ProtobufJS's verify method (it doesn't do much) by @Ekrekr in #1596
• Clean up action proto constructors in prep for proto config definitions by @Ekrekr in #1597
• Add basic actions.yaml and notebook support by @Ekrekr in #1595
• Move custom variables test to the new testing structure by @Ekrekr in #1601
• Add auto assertion database override to core test by @Ekrekr in #1602
• Move core/tasks.ts to CLI, and minor cleanup by @Ekrekr in #1598
• Add more test coverage of variables and project config overrides for Dataform Core by @Ekrekr in #1603
• Add licenses as prefixes to output bundles by @Ekrekr in #1605
• Throw a more interpretable error when a =v3 Core version by @Ekrekr in #1607
• Action builders and path utils fixes by @Ekrekr in #1606
• Prevent non bq warehouse from being set in workflow settings by @Ekrekr in #1609
• Strip notebook cell outputs by @Ekrekr in #1613
• Add version to workflow settings by @Ekrekr in #1610
• Move common target constructor methods to the action builder by @Ekrekr in #1626
• Add basic support for actions.yaml reading SQL files as operations by @Ekrekr in #1628
• Add support for declarations in action config files by @Ekrekr in #1630
• Add support for Tables to action config files by @Ekrekr in #1631
• Add action config support for the remaining SQL based action types by @Ekrekr in #1633
• Improve proto validation errors where possible by @Ekrekr in #1635
• Make filenames defined in action config files be treated as relative to the action config file by @Ekrekr in #1636
• Better errors for invalid declarations, add more test coverage by @Ekrekr in #1637
• Add an example of a workflow containing notebooks/SQL scripts, with tests by @Ekrekr in #1644
• Make the CLI use and default to workflow_settings.yaml by @Ekrekr in #1648
• Update Dataform CLI npm installs to best-effort by @Ekrekr in #1649
• Merge to v3 4 by @Ekrekr in #1654
• Remove dead resolve code by @Ekrekr in #1652
• Make main_test more concise by making filenames optional by @Ekrekr in #1651
• Remove more dead code, including navigator column descriptors and tools by @Ekrekr in #1655
• Remove warehouse option from the CLI, more tidying by @Ekrekr in #1653
• Make v3 JS API act the same as v2 by @Ekrekr in #1657
• Remove inline tables functionality by @Ekrekr in #1658
• Update proto to use a separate compiled graph, with UX review changes by @Ekrekr in #1656
• Core path tidying by @Ekrekr in #1659
• Export configs proto, bump version by @Ekrekr in #1661
• Migrate core sqlx syntax and assertions context functions tests to _test style by @Ekrekr in #1665
• Better cli errors during compilation by @Ekrekr in #1669
• Replace cli spec tests with _test format by @Ekrekr in #1670
• Improve generated workflow_settings.yaml field order by @Ekrekr in #1671
• Fix config targets by @Ekrekr in #1673
• Remove JS context from SQL files by @Ekrekr in #1672
• Lazy / stateless installation by @Ekrekr in #1675
• Don't check we are on main branch for pre-releases by @lewish in #1664
• Merge main to main_v3 5 by @Ekrekr in #1678
• Bump ip from 1.1.5 to 1.1.9 by @dependabot in #1679
• Merge v3 into main by @Ekrekr in #1680
• Bump v3 to beta by @Ekrekr in #1682

Full Changelog: 2.9.0...3.0.0-beta.0

2.9.0

What's Changed

• Bump follow-redirects from 1.15.3 to 1.15.4 by @dependabot in #1645
• Include type declarations with @dataform/core by @bmagyarkuti in #1647
• Allow 'partitionBy' and 'clusterBy' params for bigquery materialized views by @vlad-ogol in #1660

New Contributors

• @vlad-ogol made their first contribution in #1660

Full Changelog: 2.8.4...2.9.0