feat(dbt): emit table schema metadata

[rexledesma]

Summary & Motivation

Rather than relying on the user to define their table schema upfront in their dbt model properties, we can just retrieve it at run time.

On behalf of the user, we create a macro dagster__log_columns_in_relation and invoke it on the dbt project's models/seeds/snapshots. This macro emits a log with table schema information by leveraging the dbt adapter function .get_columns_in_relation (link) to retrieve column name/column data type information.

To enable this feature, the user needs to configure their @dbt_assets using their DagsterDbtTranslator. Here's a small example:

from dagster import AssetExecutionContext
from dagster_dbt import (
    DagsterDbtTranslator,
    DagsterDbtTranslatorSettings,
    DbtCliResource,
    dbt_assets,
)

from .constants import dbt_manifest_path


@dbt_assets(
    manifest=dbt_manifest_path,
    dagster_dbt_translator=DagsterDbtTranslator(
        settings=DagsterDbtTranslatorSettings(enable_table_schema_metadata=True)
    ),
)
def dbt_assets(context: AssetExecutionContext, dbt: DbtCliResource):
    yield from dbt.cli(["build"], context=context).stream()

In the future:

• We can add +meta config on individual dbt models/seeds/snapshots to prevent column information from being emitted.
• We can refactor this functionality as a separate dbt-dagster package for users to install.

How I Tested These Changes

pytest, local

Demo

schema-demo.mov

[rexledesma]

Current dependencies on/for this PR:

• master
  • PR refactor(dbt): move scaffold to subdirectory #19545
    • PR refactor(dbt): rename test_dagster_metadata to test_dagster_meta_config #19546
      • PR test(dbt): add jaffle shop project to test Dagster metadata #19547
        • PR feat(dbt): collect column metadata using the underlying dbt adapter #19712
        • PR feat(dbt): add dagster dbt package #19623
          • PR feat(dbt): collect columns metadata using the dagster dbt package #19631
            • PR docs(dbt): add reference for collecting columns metadata using the dagster dbt package #19628
        • PR feat(dbt): emit table schema metadata #19548 👈
          • PR docs(dbt): add reference to enable emitting table schema metadata #19551

This stack of pull requests is managed by Graphite.