[Bug] RuntimeError: Detected recursive loop for /usr/local/airflow/dags/dbt/dbt_venv/lib

[oliverrmaa]

Astronomer Cosmos Version

Other Astronomer Cosmos version (please specify below)

If "Other Astronomer Cosmos version" selected, which one?

1.4.3

dbt-core version

1.7.17

Versions of dbt adapters

dbt-bigquery==1.7.4
dbt-core==1.7.17
dbt-extractor==0.5.1
dbt-semantic-interfaces==0.4.4

LoadMode

DBT_LS

ExecutionMode

LOCAL

InvocationMode

SUBPROCESS

airflow version

apache-airflow==2.9.2+astro.1

Operating System

Debian GNU/Linux 11 (bullseye)

If a you think it's an UI issue, what browsers are you seeing the problem on?

No response

Deployment

Astronomer

Deployment details

We have a main production deployment in Astro Cloud which we consider as production. We also do local development via astro dev start. We have continuous deployment set up through CircleCI which deploys merged PRs to our master branch to our production deployment via astro deploy --dags. For authentication to our data warehouse (Google BigQuery) in production, we use GoogleCloudServiceAccountDictProfileMapping and for local we use ProfileConfig where our dbt profiles.yml has a hardcoded path to a service account JSON file which is at the same path for each developer.

What happened?

We noticed that we are having a RuntimeError constantly (I believe every second according to our Astro production deployment logs): Detected recursive loop when walking DAG directory /usr/local/airflow/dags: /usr/local/airflow/dags/dbt/dbt_venv/lib has appeared more than once.

We aren't sure whether this error is going to cause parts of our setup to work sub optimally or connected to any other issues we are seeing in our production environment, i.e. like in #1075, or if it will cause further issues in the future.

Relevant log output

06/25/24 15:19:42 PM	[2024-06-25T22:19:42.726+0000] {manager.py:737} INFO - Searching for files in /usr/local/airflow/dags Process ForkProcess-22795: Traceback (most recent call last): File "/usr/local/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap self.run() File "/usr/local/lib/python3.11/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/usr/local/lib/python3.11/site-packages/airflow/dag_processing/manager.py", line 241, in _run_processor_manager processor_manager.start() File "/usr/local/lib/python3.11/site-packages/airflow/dag_processing/manager.py", line 476, in start return self._run_parsing_loop() ^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/airflow/dag_processing/manager.py", line 549, in _run_parsing_loop self._refresh_dag_dir() File "/usr/local/lib/python3.11/site-packages/airflow/dag_processing/manager.py", line 738, in _refresh_dag_dir self._file_paths = list_py_file_paths(self._dag_directory) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/airflow/utils/file.py", line 298, in list_py_file_paths file_paths.extend(find_dag_file_paths(directory, safe_mode)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/airflow/utils/file.py", line 311, in find_dag_file_paths for file_path in find_path_from_directory(directory, ".airflowignore"): 

File "/usr/local/lib/python3.11/site-packages/airflow/utils/file.py", line 241, in _find_path_from_directory raise RuntimeError( RuntimeError: Detected recursive loop when walking DAG directory /usr/local/airflow/dags: /usr/local/airflow/dags/dbt/dbt_venv/lib has appeared more than once.

How to reproduce

I believe having the same set up should cause this bug to occur, note however, we have 1000+ dbt models so a project of that size may be needed to test.

Here are some relevant snippets for where our dbt_venv will live:

(1) Here is the relevant parts of our dockerfile:

FROM quay.io/astronomer/astro-runtime:11.5.0

# install dbt into a virtual environment
ARG DBT_BIGQUERY_VERSION=1.7.4
RUN python -m venv dbt_venv && source dbt_venv/bin/activate && \
    pip install --no-cache-dir dbt-bigquery==$DBT_BIGQUERY_VERSION && deactivate

(2) Here is our ExecutionConfig

ExecutionConfig(
            execution_mode=ExecutionMode.LOCAL,
            dbt_executable_path=f"{os.environ['AIRFLOW_HOME']}/dbt_venv/bin/dbt",
        )

Anything else :)?

Here is a screenshot of our production deployment logs in Astro UI where we found the bug, as you can see the issue is continuously happening:

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Contact Details

oma@forhims.com

[dosubot]

To resolve the RuntimeError indicating a recursive loop due to the dbt_venv/lib appearing more than once, you should ensure that the dbt_venv directory is excluded from the DAGs directory. Here’s how you can do it:

1. Move the dbt_venv Directory:
   Move the dbt_venv directory outside of the DAGs directory. For example, place it in a different location such as /usr/local/airflow/dbt_venv.

2. Update the dbt_executable_path:
   Update the dbt_executable_path in your DAG configuration to point to the new location of the dbt_venv directory.

Here’s an example of how you can update your ExecutionConfig in the DAG configuration:

import os
from datetime import datetime
from cosmos import DbtDag, ProjectConfig, ProfileConfig, ExecutionConfig
from cosmos.profiles import GoogleCloudServiceAccountDictProfileMapping

profile_config = ProfileConfig(
    profile_name="default",
    target_name="dev",
    profile_mapping=GoogleCloudServiceAccountDictProfileMapping(
        conn_id="google_cloud_default",
        profile_args={"schema": "public"},
    ),
)

execution_config = ExecutionConfig(
    execution_mode=ExecutionMode.LOCAL,
    dbt_executable_path=f"{os.environ['AIRFLOW_HOME']}/dbt_venv/bin/dbt",
)

my_cosmos_dag = DbtDag(
    project_config=ProjectConfig(
        "/usr/local/airflow/dags/my_dbt_project",
    ),
    profile_config=profile_config,
    execution_config=execution_config,
    schedule_interval="@daily",
    start_date=datetime(2023, 1, 1),
    catchup=False,
    dag_id="my_cosmos_dag",
    default_args={"retries": 2},
)

By ensuring that the dbt_venv directory is outside of the DAGs directory, you can avoid the recursive loop issue. Additionally, you can use MWAA's startup script to install dbt into a virtual environment, as outlined in the MWAA documentation. This ensures that the virtual environment is properly set up before the scheduler and webserver start ^[1][2].

To continue talking to Dosu, mention @dosu.

[tatiana]

Hi @oliverrmaa , thanks for the detailed description, this is very helpful!

Are you storing the dbt project files in /usr/local/airflow/dags/dbt/?

Is /usr/local/airflow/dags/dbt/dbt_venv/ a Python virtualenv containing the dbt-core installation?

Would it be possible to not have dbt_venv inside the dbt or Airflow DAGs folder? This should solve the issue.

[tatiana]

@oliverrmaa the root cause is that the Airflow DAGs folder contains a dbt_venv Python virtualenv folder that contains recursive loops. When this happens, the Airflow scheduler cannot parse the DAGs and cannot orchestrate workflows - it breaks.

Although Cosmos is not introducing this issue, we spiked on the implementation of two functions and a command line that could help clean up the DAGs directory during the deployment, which we shared today: #1115

We made this change available in an Alpha release: https://github.com/astronomer/astronomer-cosmos/releases/tag/astronomer-cosmos-v1.6.0a3. You could try it out installing Cosmos using:

pip install astronomer-cosmos==v1.6.0a3

Alec mentioned that it was unclear how this folder was being created since it was not part of the Github repo. Therefore, this tooling would be less useful. For this reason, I'll close this PR without merging it with the main. Currently, these features will not be part of a stable Cosmos release. We can reopen this in the future if you or other community members find value on these methods.

Today, we also discussed that currently, your team has a few different ways of deploying to Astro, including:
a) From the CI/CD
b) Some developers use Astro CLI from their local machines

During the call, we reviewed (a) your Astro Dockerfile and confirmed that it is not creating the dbt_venv directory inside the DAGs directory.

Therefore, you're facing the problem because of deployments derived from local developers' computers (b). As we discussed, the ideal scenario would be for the virtualenv not to be created inside the dbt folder.

During the call, Alec mentioned he had been advised that files in the .gitignore were not deployed as part of the Astro deployment. I confirmed with the Astro CLI team today, and I'm afraid this information is inaccurate.

As of Astro CLI 1.27.1:
i) The Astro CLI does not use .gitignore in any way
ii) When performing image-based deployments, the Astro CLI takes into account .dockerignore
iii) When performing dag-based deployments (--dags), the Astro CLI does not use .dockerignore

There is a solution to avoid the problem from happening again - as long as the path of the dbt_venv is always in the exact location in the DAGs folder:

• Introduce a .airflowignore file inside your dags folder. It should contain the paths to all possible relative paths of the dbt_venv directory inside the DAGs folder. Example:

dbt_venv
dbt/dbt_venv

By doing this, Airflow should ignore these paths containing recursive loops and be able to parse DAGs, as explained in the docs:

• https://www.astronomer.io/docs/astro/cli/airflowignore
• https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/dags.html#airflowignore

However, suppose developers create Python virtual environments with recursive loops in different paths in the DAGs folder. In that case, the issue will happen again: DAGs will not run on schedule. For this reason, it is essential to review and align with the team on the current deployment processes from local dev machines.

Future actions that are outside of the scope of Cosmos could be:

• To change Astro CLI to check if the /dags folder has recursive loops before any deployment - and erroring at the client-side: Before deploying (both image-based an dags-based), check for recursive loops astro-cli#1682
• To change Airflow for it to stop raising a RuntimeError when it finds a recursive loop: RuntimeError "Detected recursive loop when walking DAG directory" apache/airflow#40980