Overview

Welcome to Astronomer! This project was generated after you ran 'astro dev init' using the Astronomer CLI. This readme describes the contents of the project, as well as how to run Apache Airflow on your local machine.

Project Contents

test_project

description: An Astro project for Test team in AI Platform Engineering Team: Test team Author: Test team (test.team@paychex.com)

Your Astro project contains the following files and folders:

• dags: This folder contains the Python files for your Airflow DAGs. By default, this directory includes one example DAG:
  • example_astronauts: This DAG shows a simple ETL pipeline example that queries the list of astronauts currently in space from the Open Notify API and prints a statement for each astronaut. The DAG uses the TaskFlow API to define tasks in Python, and dynamic task mapping to dynamically print a statement for each astronaut. For more on how this DAG works, see our Getting started tutorial.
• Dockerfile: This file contains a versioned Astro Runtime Docker image that provides a differentiated Airflow experience. If you want to execute other commands or overrides at runtime, specify them here.
• include: This folder contains any additional files that you want to include as part of your project. It is empty by default.
• packages.txt: Install OS-level packages needed for your project by adding them to this file. It is empty by default.
• requirements.txt: Install Python packages needed for your project by adding them to this file. It is empty by default.
• plugins: Add custom or community plugins for your project to this file. It is empty by default.
• airflow_settings.yaml: Use this local-only file to specify Airflow Connections, Variables, and Pools instead of entering them in the Airflow UI as you develop DAGs in this project.

Deploy Your Project Locally

Important: Local Development Setup

Before starting Airflow locally for the first time, run:

./setup_project.sh

This script will:

1. Optionally configure Azure Key Vault integration

How It Works:

• Dockerfile: Uses conditional logic to work for both local and cloud environments
  • Local development: Automatically detects and installs bootstrap/paychex.pem certificate
  • Cloud deployment: Uses vanilla Astronomer runtime (certificate file not present)
• bootstrap/paychex.pem: Place your Paychex certificate here for local development (gitignored)
• Same Dockerfile works everywhere - no manual switching needed!

Starting Airflow

After running setup_project.sh, start Airflow on your local machine by running:

astro dev start

This command will spin up five Docker containers on your machine, each for a different Airflow component:

• Postgres: Airflow's Metadata Database
• Scheduler: The Airflow component responsible for monitoring and triggering tasks
• DAG Processor: The Airflow component responsible for parsing DAGs
• API Server: The Airflow component responsible for serving the Airflow UI and API
• Triggerer: The Airflow component responsible for triggering deferred tasks

When all five containers are ready the command will open the browser to the Airflow UI at http://localhost:8080/. You should also be able to access your Postgres Database at 'localhost:5432/postgres' with username 'postgres' and password 'postgres'.

Note: If you already have either of the above ports allocated, you can either stop your existing Docker containers or change the port.

Configure Azure Key Vault (Optional)

This project supports Azure Key Vault integration for managing Airflow connections and variables securely. This allows your local Airflow instance to pull secrets from Azure Key Vault, mimicking the production environment.

Prerequisites

1. Azure CLI installed and configured (az --version to verify)
2. Access to an Azure Key Vault (created via Cortex Workflow)
3. Azure login credentials (az login to authenticate)
4. Permissions to read secrets from your Key Vault (Key Vault Secrets User role)

Quick Setup

Run the Azure Key Vault configuration script:

./bootstrap/setup_azure_keyvault.sh

This script will:

• Verify Azure CLI and Astro CLI installations
• Check your Azure login status
• Prompt for your Key Vault name
• Validate access to the Key Vault
• Configure the .env file with proper Azure Key Vault backend settings
• Optionally add the required Azure provider package to requirements.txt

Note: The script is also automatically offered during the initial ./setup_project.sh run.

Manual Configuration

If you prefer to configure manually, add the following to your .env file:

AIRFLOW__SECRETS__BACKEND=airflow.providers.microsoft.azure.secrets.key_vault.AzureKeyVaultBackend
AIRFLOW__SECRETS__BACKEND_KWARGS={"connections_prefix":"airflow-connections","variables_prefix":"airflow-variables","vault_url":"https://YOUR-VAULT-NAME.vault.azure.net/","tenant_id":"bcc529c5-dfce-4f97-b44f-debd50891d83"}

Replace YOUR-VAULT-NAME with your actual Key Vault name.

Also ensure apache-airflow-providers-microsoft-azure>=11.0.0 is in your requirements.txt.

Authentication

Local development uses Service Principal authentication for Docker environments.

The setup script automatically:

• Creates a service principal for your user (e.g., sp-airflow-local-dev-youruser)
• Grants it permissions to read secrets from your Key Vault
• Stores the client_id and client_secret in .env

Before starting Airflow, ensure you're logged in to Azure:

az login

The script will handle the rest!

Storing Secrets in Key Vault

Secrets in Azure Key Vault should follow naming conventions:

• Connections: airflow-connections-<connection_id>
• Variables: airflow-variables-<variable_name>

Example: To create an Airflow connection named my_postgres_db:

az keyvault secret set \
  --vault-name YOUR-VAULT-NAME \
  --name airflow-connections-my-postgres-db \
  --value "postgresql://user:password@host:5432/database"

Verifying the Setup

After configuration:

1. Restart Airflow (if already running):

   astro dev restart

2. Check the Airflow UI → Admin → Connections - you should see connections from Key Vault

3. Check logs for any authentication errors:

   astro dev logs -f webserver

Troubleshooting

403 Forbidden Errors:

• Ensure you have the "Key Vault Secrets User" role on the Key Vault
• Ask your administrator to grant permissions:

  az keyvault set-policy --name YOUR-VAULT-NAME \
    --upn $(az account show --query user.name -o tsv) \
    --secret-permissions get list

Authentication Failures:

• Run az login again
• Verify you're logged into the correct Azure tenant
• Check that the tenant ID in .env is correct

Secrets Not Appearing:

• Verify secret naming follows the prefix convention
• Check that prefixes in .env match your Key Vault structure
• Ensure secrets exist: az keyvault secret list --vault-name YOUR-VAULT-NAME

Deploy Your Project to Astronomer

Dockerfile Management

The workflow is simple - the Dockerfile uses conditional logic:

• Dockerfile (committed to repo): Uses FROM astrocrpublic.azurecr.io/runtime:3.1-10
  • Conditionally installs Paychex certificate if bootstrap/paychex.pem exists
  • Works seamlessly for both local development and cloud deployment
• Local development: Place bootstrap/paychex.pem → certificate automatically installed
• Cloud deployment: Certificate file not present (gitignored) → skips certificate installation

No manual switching, no separate images, no restore scripts needed!

Deploying

If you have an Astronomer account, pushing code to a Deployment on Astronomer is simple. For deploying instructions, refer to Astronomer documentation: https://www.astronomer.io/docs/astro/deploy-code/

Workflow Summary

1. Clone repo → Dockerfile with conditional certificate installation
2. Run ./setup_project.sh → Configure Azure Key Vault (optional)
3. Place certificate at bootstrap/paychex.pem for local development
4. Run astro dev start → Dockerfile detects certificate and installs it
5. Develop locally with certificates and dependencies
6. Commit and push → Certificate is gitignored, not pushed to cloud
7. Astronomer cloud builds using same Dockerfile → skips certificate installation

Contact

The Astronomer CLI is maintained with love by the Astronomer team. To report a bug or suggest a change, reach out to our support.