Python models: next steps

[lostmygithubaccount]

At dbt Labs, we're very excited Python models are out in the 1.3 beta and on track for launch at Coalesce. The purpose of this discussion is to talk about the next steps beyond 1.3.

Before jumping into this discussion, first read:

• dbt Python models · Discussion #5261 · dbt-labs/dbt-core (github.com)
• Python models | dbt Docs (getdbt.com)

Overview

Python models in dbt today are a first step. As noted in the original discussion, they open pandoras box. One obvious domain we'll continue to investigate and increase support for is data science and the machine learning lifecycle management -- dbt is well-positioned for this with Python models as a first step.

There are a number of open product decisions to work through. We hope by opening these discussions here for your input, we can come to conclusions on those decisions with the community.

Planned discovery work

We've opened two relevant, focused discussions so far:

• Python models: the pandas problem (and a possible solution) #5738
• Python models: package, artifact/object storage, and UDF management in dbt #5741

In addition to those topics, we plan to investigate and likely open further discussions for:

• MLflow support in dbt for ML model tracking and cross-platform model registry
• semantic layer for Python: inputs and outputs
• dbt Python package ecosystem
• tests written natively in Python models
• docs written natively in Python models
• interactive Python support

If any of these are particularly interesting to you, or there's something we're missing, let us know! We plan to investigate these over coming months and continue to provide updates on our roadmap.

[the-serious-programmer]

Hey! I couldn't find this mentioned by someone else, so I will list it here.

Currently we run DBT in Apache Airflow (with the dbt-bigquery adapter). Once DBT is done Airflow makes some HTTP calls to a python API, containing statistical functions that we need to calculate some results.

When reading up about the python models, our team got initially quite enthusiastic, as this has great potential for us to embed these statistical functions into the datapipeline.

My points are:

• Please be aware that embedding statistical functions is a use case for us
• To be able to make this use case possible, being able to use private packages with DBT is a must

[aaronsteers]

Cross-posted from...

• dbt Python models #5261 (comment)

I'm curious what the thinking is regarding dynamic dependency and non-deterministic ref() statements in a Python model. Is there a plan for handling them as of now? Said simply, on a first pass, the Python model declares it needs ref("table_1") but on a subsequent pass it also asks for ref("table_2") and ref("table_3") - one or both of which aren't ready yet.

I can think of a few solutions, like a two-pass approach, a separate codepath in the python model itself, pausing/restarting execution if a requested input isn't ready, etc.

Is this already handled and/or is perhaps already in scope for current/future iterations?

[ValdarT]

Not specific to python but I think it makes it even more likely that people would like to connect to external sources of data so secret variables in models (that would be hidden from the logs) becomes even more wanted feature to securely provide the connection details. I know that source nodes that actually import data are not the originally intended use case but it makes a ton of sense to support in my opinion, especially with python support added.

[nitinbhojwani]

tests written natively in Python models would help with more complex data quality checks.

[0xj7r]

Apologies if this has been asked elsewhere; is there a timeline for dbt-py to be enabled for Redshift?

[lostmygithubaccount]

hi @0xj7r, asking internally for an update but I don't think we have any committed timeline yet. you can follow along with this issue to stay up to date on when it's implemented and merged: dbt-labs/dbt-redshift#204

[0xj7r]

Awesome, thanks!

[JackWolverson]

Hello, couldnt see an issue noted anywhere, can see above there is an open issue for python models in dbt-redshift, is there a similar issue open for postgres ?

[MeronHaileTesfazion]

Hi,
I do not see anyone else mentioning this particular feature, but my understanding now is that there is no support for inbuilt logging through the python module logging. I think it would be beneficial to be able to view stdout logging in the dbt logs as well. Both for development purposes and also while running in production.
Great work!

[Pawel-Drabczyk]

Hi,

is there any news about adding a code re-usage option to DBT Python models? My team wants to migrate some Python functions (currently run on Kubernetes) to DBT Python models and we would really appreciate this feature.

[Sandhiya-Munishwaran]

Hi,

I'm looking to run more complex SQL statements in my Python model referring to other dbt SQL models. I want to reference these models directly, similar to how it's done with Jinja, without creating DataFrames from them. Is there an existing way to do this? If not, I think this would be a valuable feature to consider for the next release.