Anyone using with MWAA 2.6.3 or later?

[satar]

We've been using airflow-dbt-python with MWAA for well over a year now and it worked out very well but took quite a bit of effort to get just right. We are using Airflow 2.2.2 currently with requirements.txt as follows:

# Using this when running mwaa-local-runner or an mwaa environment combined with airflow-dbt-python==0.15.2 works
apache-airflow-providers-amazon==6.0.0

# Installs dbt core 1.2.3 and snowflake plugin adapter 1.2.0
airflow-dbt-python==0.15.2
airflow-dbt-python[snowflake]==0.15.2

The problem is that Amazon will be ending support for v2.2.2 on June 27, 2024 thus we are forced to upgrade to a newer version. I made a futile attempt to upgrade to v2.5.1 several months back and spent over a day so far attempting to upgrade to v2.8.1 and abandoning this to the last python 3.10 supported version v2.6.3. Is anyone in the community using MWAA + dbt + Snowflake dbt adapter and an Airflow version > 2.2.2?

[marcusintrohive]

@satar we have been using Airflow 2.8.1 dbt and snowflake.

Something we needed to do with our latest upgrade is fork the dbt-core repo and make a few changes in order satisfy the pip contraints.

We are using Airflow 2.8.1 and dbt 1.7.11.

[satar]

@marcusintrohive this sounds very promising. I also reached out to AWS support and they suggested creating a startup script instead of using requirements.txt -- see: https://docs.aws.amazon.com/mwaa/latest/userguide/samples-dbt.html#samples-dbt-dependencies
Although I haven't tried this technique with an MWAA Environment instance deployed to AWS I have made several various attempts using mwaa-local-runner v2.8.1. I had success with installing:

pip3 install dbt-core==1.7.8
pip3 install airflow-dbt-python[snowflake]

But if I don't deactivate the virtual machine at the end of the startup script it causes other issues and if I deactivate it then I need to somehow activate when running dbt dags, which I am not sure is even possible.

It would be nice to try your technique but not sure how to go about creating a patched version of dbt-core and reference it. Are you just putting everything in a plugins zip file?

UPDATE: I attempted to add the mwaa-base-providers-requirements to startup script and it work! Well, it works so far when running in aws-mwaa-local-runner on my mac but haven't tried in an AWS Environment yet. Will do that tomorrow.

UPDATE: Sadly, this approach didn't work when deployed to an MWAA Environment within AWS... @marcusintrohive if you would be willing to share more details on your approach I would be extremely grateful. Otherwise, what both AWS support and dbt support suggest is to use the startup script to generate a virtualenv for dbt + snowflake and then use BashOperators for interacting with dbt... for example:

	with DAG(dag_id=DAG_ID, schedule_interval=None, catchup=False, start_date=days_ago(1)) as dag:
	    cli_command = BashOperator(
	        task_id="bash_command",
	        bash_command="source /usr/local/airflow/python3-virtualenv/dbt-env/bin/activate;\
							cp -R /usr/local/airflow/dags/dbt /tmp;\
							echo 'listing project files:';\
							ls -R /tmp;\
							cd /tmp/dbt/mwaa_dbt_test_project;\
							/usr/local/airflow/python3-virtualenv/dbt-env/bin/dbt run --project-dir /tmp/dbt/mwaa_dbt_test_project --profiles-dir ..;\
							cat /tmp/dbt_logs/dbt.log;\
							rm -rf /tmp/dbt/mwaa_dbt_test_project"
	    )

FINAL UPDATE: We got around this by using a modified constraints file.

[sandippani]

Hi @satar I am too struggling with same , no luck yet.
Can you share what changes you made to constraint file pertaining to this issue?