dagster-dbt: DagsterInvariantViolationError in dbt_assets with valid manifest.json

[traanhn]

What's the issue?

Summary

After upgrading from Dagster 1.9.6 to 1.9.7, the dbt_assets function fails with the error:

dagster._core.errors.DagsterInvariantViolationError: Unexpected 'None' output value. If a 'None' value is intentional, set the output type to None by adding return type annotation '-> None'.

This issue does not occur with Dagster 1.9.6 or smaller version .

Dagster Version

• Dagster Version: 1.9.7 (also tested 1.9.8, same issue)
• dagster-dbt Version: 0.25.7
• DBT Version: 1.8.7

Observed Behavior

The job fails immediately after yielding the first output from dbt.cli(...).stream(). The STEP_FAILURE logs show that Dagster encounters a None value unexpectedly.

Workaround

Reverting to Dagster 1.9.6 resolves the issue.

Environment Information

• OS: macOS 15.2 ARM (M1)
• Python Version: 3.12.0
• Dagster Version: 1.9.7 and 1.9.8
• Dagster-DBT Version: 0.25.7 and 0.25.8
• DBT Version: 1.8.7

What did you expect to happen?

Expected Behavior

The job should run successfully without errors, as it does with Dagster 1.9.6.

How to reproduce?

Steps to Reproduce

1. Create a dbt_assets function with a valid manifest.json:

@dbt_assets(
    manifest=env_config.dbt_manifest_path,
    dagster_dbt_translator=CustomDagsterDbtTranslator(),
    io_manager_key="snowflake_analytics_io_manager"
)
def all_dbt_assets(context: AssetExecutionContext, dbt: DbtCliResource, config: DbtConfig):
    dbt_build_args = ["build"]
    if config.full_refresh:
        dbt_build_args += ["--full-refresh"]

    dbt_build_invocation = dbt.cli(dbt_build_args, context=context)
    yield from dbt_build_invocation.stream()

2. Run the job:

   dagster job execute -f my_job.py

3. Observe the error:

   dagster._core.errors.DagsterInvariantViolationError: Unexpected 'None' output value. If a 'None' value is intentional, set the output type to None by adding return type annotation '-> None'.
   

Dagster version

1.9.7

Deployment type

Local

Deployment details

No response

Additional information

Testing with GKE staging environment generates the same issue with version 1.9.7 for dagster-dbt= 0.25.7
Revert to 0.25.6 then it works in both local and GKE.

It seems there are some changes in dbt_assets decorator in version 1.9.7 which has caused this unexpectedly

Message from the maintainers

Impacted by this issue? Give it a 👍! We factor engagement into prioritization.

[cdchan]

We've been seeing this too after upgrading to 1.9.8. It looks like it might be related to #26614 which removed the explicit AssetOut with type Nothing

[mgierada]

I also observe this issue as of 1.9.11

[mgierada]

Hey @traanhn @cdchan , I spent some time to debug this. Here's what I did to make it work, though, I still believe a proper fix would be required. The code below can still raise StopIteration but it work okay for me.

@dbt_assets(
    manifest=env_config.dbt_manifest_path,
    dagster_dbt_translator=CustomDagsterDbtTranslator(),
    io_manager_key="snowflake_analytics_io_manager"
    ) -> t.Generator[
        t.Union[Output, AssetMaterialization, AssetObservation, AssetCheckResult],
        None,
        None,
    ]:

def all_dbt_assets(context: AssetExecutionContext, dbt: DbtCliResource, config: DbtConfig):
    dbt_build_args = ["build"]
    if config.full_refresh:
        dbt_build_args += ["--full-refresh"]

    dbt_build_invocation = dbt.cli(dbt_build_args, context=context)
    cli_iterator = dbt_build_invocation.stream()

    if cli_iterator.__next__() is not None:
        yield from cli_iterator

I've been inspired by this issue #24298

[mgierada]

The problem with the code above is that AssetMaterialization won't be recorded and one won't be able to figure out when asset was materialized.