Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Dagster 1.9.7 fails to initialize if a dbt model has unit tests #27007

Open
stiabh opened this issue Jan 10, 2025 · 2 comments · May be fixed by #27052
Open

Dagster 1.9.7 fails to initialize if a dbt model has unit tests #27007

stiabh opened this issue Jan 10, 2025 · 2 comments · May be fixed by #27052
Labels
integration: dbt Related to dagster-dbt type: bug Something isn't working

Comments

@stiabh
Copy link

stiabh commented Jan 10, 2025 •
edited
Loading

What's the issue?

Dagster will fail to initialize if any dbt model has an associated unit test, due to a slight change to unit test config in dbt 1.9. (With the release of dagster-dbt 0.25.7, dagster supports dbt 1.9).

dbt 1.9 added support for disabling unit tests: dbt-labs/dbt-core#10831

In doing so, the config dict of the "unit_tests" attribute in the dbt manifest is expected to be dot-accessible: dbt-labs/dbt-core@81067d4#diff-b4756f21e0fa65b77bb36828994d3a758360b5acd05bbb9526b5bfe399194617

Dicts are not dot-accessible in Python by default, and dagster-dbt handles this with the _DictShim class in utils.py when parsing the dbt manifest. The problem that the unit_tests block in the manifest has to be unpacked into a dbt.contracts.graph.nodes.UnitTestDefinition object. As the element is unpacked, the config attibute is not _DictShim, but a normal dict instead, and thereby not dot-accessible. See https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-dbt/dagster_dbt/utils.py#L54

Solution

The Manifest object produced in select_unique_ids_from_manifest() has to ensure that its attribute unit_tests.config is dot accessible.

Error message returned by dagster dev

AttributeError: 'dict' object has no attribute 'enabled'

Stack Trace:
  File "/home/stiabh/repos/let/.venv/lib/python3.10/site-packages/dagster/_core/errors.py", line 287, in user_code_error_boundary
    yield
  File "/home/stiabh/repos/let/.venv/lib/python3.10/site-packages/dagster/_grpc/server.py", line 258, in __init__
    loadable_targets = get_loadable_targets(
  File "/home/stiabh/repos/let/.venv/lib/python3.10/site-packages/dagster/_grpc/utils.py", line 50, in get_loadable_targets
    else loadable_targets_from_python_module(module_name, working_directory)
  File "/home/stiabh/repos/let/.venv/lib/python3.10/site-packages/dagster/_core/workspace/autodiscovery.py", line 32, in loadable_targets_from_python_module
    module = load_python_module(
  File "/home/stiabh/repos/let/.venv/lib/python3.10/site-packages/dagster/_core/code_pointer.py", line 134, in load_python_module
    return importlib.import_module(module_name)
  File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/home/stiabh/repos/let/data/data_pipeline/__init__.py", line 12, in <module>
    from . import assets
  File "/home/stiabh/repos/let/data/data_pipeline/assets/__init__.py", line 3, in <module>
    from . import (
  File "/home/stiabh/repos/let/data/data_pipeline/assets/dbt.py", line 48, in <module>
    @dbt_assets(
  File "/home/stiabh/repos/let/.venv/lib/python3.10/site-packages/dagster/_core/decorator_utils.py", line 223, in wrapped_with_context_manager_fn
    return fn(*args, **kwargs)
  File "/home/stiabh/repos/let/.venv/lib/python3.10/site-packages/dagster_dbt/asset_decorator.py", line 305, in dbt_assets
    specs, check_specs = build_dbt_specs(
  File "/home/stiabh/repos/let/.venv/lib/python3.10/site-packages/dagster_dbt/asset_utils.py", line 806, in build_dbt_specs
    selected_unique_ids = select_unique_ids_from_manifest(
  File "/home/stiabh/repos/let/.venv/lib/python3.10/site-packages/dagster_dbt/utils.py", line 115, in select_unique_ids_from_manifest
    selector = graph_selector.NodeSelector(graph, manifest)
  File "/home/stiabh/repos/let/.venv/lib/python3.10/site-packages/dbt/graph/selector.py", line 52, in __init__
    graph_members = {
  File "/home/stiabh/repos/let/.venv/lib/python3.10/site-packages/dbt/graph/selector.py", line 53, in <setcomp>
    unique_id for unique_id in self.full_graph.nodes() if self._is_graph_member(unique_id)
  File "/home/stiabh/repos/let/.venv/lib/python3.10/site-packages/dbt/graph/selector.py", line 177, in _is_graph_member
    return unit_test.config.enabled

What did you expect to happen?

No response

How to reproduce?

Install dbt-dagster 0.25.7 and dbt >=1.9.0 and initialize dagster with a dbt project containing at least one unit test.

Dagster version

1.9.7

Deployment type

None

Deployment details

No response

Additional information

No response

Message from the maintainers

Impacted by this issue? Give it a 👍! We factor engagement into prioritization.
@stiabh stiabh added the type: bug Something isn't working label Jan 10, 2025
@stiabh stiabh changed the title Dagster fails to start if a dbt model has unit tests, after upgrading to dbt 1.9 dbt-dagster crashes fails to start if a dbt model has unit tests, after upgrading to dbt 1.9 Jan 10, 2025
@stiabh stiabh changed the title dbt-dagster crashes fails to start if a dbt model has unit tests, after upgrading to dbt 1.9 dbt-dagster crashes on dbt models with unit tests, after upgrading to dbt 1.9 Jan 10, 2025
@stiabh stiabh changed the title dbt-dagster crashes on dbt models with unit tests, after upgrading to dbt 1.9 Dagster fails to start if a dbt model has unit tests, after upgrading to dbt 1.9 Jan 10, 2025
@stiabh stiabh changed the title Dagster fails to start if a dbt model has unit tests, after upgrading to dbt 1.9 Dagster 1.9.7 fails to start if a dbt model has unit tests Jan 10, 2025
@stiabh stiabh changed the title Dagster 1.9.7 fails to start if a dbt model has unit tests Dagster 1.9.7 fails to initialize if a dbt model has unit tests Jan 10, 2025
@kang8 kang8 linked a pull request Jan 13, 2025 that will close this issue
Open
kang8 added a commit to kang8/dagster that referenced this issue Jan 13, 2025
@kang8
[dagster-dbt] Refactor UnitTestDefinition instantiation in utils.py t…
2569dbe 
…o use from_dict method

Verified `test_dbt_with_unit_tests()` passes with both "fqn:*" and "tag:test" selectors.

Updated the instantiation of UnitTestDefinition in the __getattr__
method to utilize the from_dict method instead of the constructor. This
change enhances the clarity and consistency of the code when handling
unit test nodes from the manifest JSON.

Fixed: dagster-io#27007
@garethbrickman garethbrickman added the integration: dbt Related to dagster-dbt label Jan 13, 2025
@garethbrickman garethbrickman mentioned this issue Jan 13, 2025
Closed
kang8 added a commit to kang8/dagster that referenced this issue Jan 14, 2025
@kang8
[dagster-dbt] Refactor UnitTestDefinition instantiation in utils.py t…
1559c25 
…o use from_dict method

Verified `test_dbt_with_unit_tests()` passes with both "fqn:*" and "tag:test" selectors.

Updated the instantiation of UnitTestDefinition in the __getattr__
method to utilize the from_dict method instead of the constructor. This
change enhances the clarity and consistency of the code when handling
unit test nodes from the manifest JSON.

Fixed: dagster-io#27007
@jakubczaplicki
Copy link

Is there a workaround this issue ? Is anyone actively working on a fix ?

@garethbrickman
Copy link
Contributor

@jakubczaplicki A fix is in development: #27052

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
integration: dbt Related to dagster-dbt type: bug Something isn't working
Projects
None yet
Milestone
No milestone
3 participants
@jakubczaplicki @garethbrickman @stiabh