init_log_folder raises PermissionError when base_log_folder is on a mounted filesystem

[hkc-8010]

Apache Airflow version

3.1.7

Operating System

Linux (OpenShift / Kubernetes)

Deployment

Docker / Kubernetes

Deployment details

Airflow 3.1.7 running on Kubernetes (OpenShift) with task logs configured to write to a network-mounted filesystem (NFS) via AIRFLOW__LOGGING__BASE_LOG_FOLDER.

What happened

After setting AIRFLOW__LOGGING__BASE_LOG_FOLDER to a path on a mounted volume (e.g., /mnt/nfs-share/airflow/logs), any Airflow CLI command — including airflow db migrate — crashes with a PermissionError during settings.initialize():

Traceback (most recent call last):
  File "/home/airflow/.local/bin/airflow", line 3, in <module>
    from airflow.__main__ import main
  File "/home/airflow/.local/lib/python3.11/site-packages/airflow/__init__.py", line 79, in <module>
    settings.initialize()
  File "/home/airflow/.local/lib/python3.11/site-packages/airflow/settings.py", line 698, in initialize
    LOGGING_CLASS_PATH = configure_logging()
  File "/home/airflow/.local/lib/python3.11/site-packages/airflow/logging_config.py", line 128, in configure_logging
    return init_log_folder(
  File "/home/airflow/.local/lib/python3.11/site-packages/airflow/_shared/logging/structlog.py", line 579, in init_log_folder
    parent.mkdir(mode=new_folder_permissions, exist_ok=True)
  File "/usr/python/lib/python3.11/pathlib.py", line 1116, in mkdir
    os.mkdir(self, mode)
PermissionError: [Errno 13] Permission denied: '/mnt/nfs-share'

The root cause is in init_log_folder() (introduced in #55431):

def init_log_folder(directory, new_folder_permissions):
    directory = Path(directory)
    for parent in reversed(Path(directory).parents):
        parent.mkdir(mode=new_folder_permissions, exist_ok=True)
    directory.mkdir(mode=new_folder_permissions, exist_ok=True)

This iterates through all parent directories starting from /, calling mkdir() on each one. For a path like /mnt/nfs-share/airflow/logs, the iteration calls mkdir on /, /mnt, /mnt/nfs-share, /mnt/nfs-share/airflow, and finally the target directory.

When the mount point at /mnt/nfs-share does not yet exist (e.g., the volume hasn't been mounted yet during an init container running airflow db migrate), os.mkdir('/mnt/nfs-share') fails with PermissionError because the airflow user cannot create directories in /mnt. The exist_ok=True parameter only catches FileExistsError, not PermissionError, so the error propagates and crashes the entire Airflow startup.

Even when the mount IS available, this pattern is fragile — it attempts to mkdir on system-level directories that the airflow user should never need to create.

Additionally, there is no try/except around the init_log_folder call in configure_logging() (logging_config.py:158), so the PermissionError crashes all Airflow CLI commands, including airflow db migrate which doesn't actually need the log folder.

What you think should happen instead

init_log_folder should not crash Airflow when parent directories of the log folder cannot be created. Several possible improvements:

1. Don't iterate through system-level parents: Replace the explicit parent loop with directory.mkdir(parents=True, exist_ok=True), which will stop at the first existing parent directory and only create subdirectories below it. This avoids calling mkdir on paths like /, /mnt, etc.

2. Catch PermissionError / OSError on parents that already exist: When calling mkdir on a parent directory, if the call raises PermissionError, check whether the directory already exists (parent.is_dir()) and skip it if so. Only re-raise if the directory genuinely doesn't exist.

3. Wrap the init_log_folder call in a try/except: At minimum, commands like airflow db migrate should not crash because the log folder cannot be created. A warning would be more appropriate.

How to reproduce

1. Use the official Airflow Docker image (3.1.7)
2. Set AIRFLOW__LOGGING__BASE_LOG_FOLDER=/mnt/nfs-share/airflow/logs
3. Do not mount any volume at /mnt/nfs-share
4. Run airflow db migrate (or any airflow CLI command)
5. Observe the PermissionError crash

This also reproduces when the base log folder is on a network mount that is not yet available at the time Airflow starts (common in Kubernetes init containers).

Anything else

Context on why this surfaces now: In Airflow 3.x, base_log_folder was moved from the [core] section to the [logging] section. Users migrating from AIRFLOW__CORE__BASE_LOG_FOLDER to AIRFLOW__LOGGING__BASE_LOG_FOLDER are newly affected because the old env var silently did not propagate to configure_logging() (which reads from [logging]), so the default local path was used and init_log_folder never encountered a mounted path.

The init_log_folder function was introduced in PR #55431 as part of the structlog migration. The previous Airflow 2.x code path for log folder creation did not iterate through all parent directories in this way.

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct