Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

dbt-redshift lineage metadata for external databases like awsdata catalog #973

Open
2 tasks done
bhawesh490 opened this issue Jan 10, 2025 · 0 comments
Open
2 tasks done
Labels
bug Something isn't working triage

Comments

@bhawesh490
Copy link

Is this a new bug in dbt-redshift?

  • I believe this is a new bug in dbt-redshift
  • I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

Hi Team
I am working on a use case where i have to create dbt models on redshift.
On top of these models we have to build data lineage graph on marquez.
My dbt model has 2 tables (one staging table and other one fact table)
Staging table-it queries data from aws data catalog like this

`
{{ config(materialized='view',bind=False) }}

with source_data as (

SELECT book,hfm_code,entity
FROM
"awsdatacatalog"."database_name"."table_name" limit 1000

)

select *
from source_data

fact table model looks like this
{{ config(materialized='table') }}

SELECT book,hfm_code
FROM
{{ ref('staging_fa_capital_local_af') }}
where entity = '5001'

`
my expectation is that i should see 3 nodes in the lineage graph

1st node: representing source as glue table
2nd node:representing the staging table
3rd node:representing the fact table

However lineage medata data does not provide any information about source coming from external database like awsdata catalog
Lineage event generated is:

image

Expected Behavior

my expectation is that i should see 3 nodes in the lineage graph

1st node: representing source as glue table
2nd node:representing the staging table
3rd node:representing the fact table

Steps To Reproduce

my expectation is that i should see 3 nodes in the lineage graph

1st node: representing source as glue table
2nd node:representing the staging table
3rd node:representing the fact table

Relevant log output

No response

Environment

- OS:
- Python:
- dbt-core:
- dbt-redshift:

Additional Context

No response

@bhawesh490 bhawesh490 added bug Something isn't working triage labels Jan 10, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working triage
Projects
None yet
Development

No branches or pull requests

1 participant