airflow-dags

In this repository, we've shared all dags for TogetherCrew. The dags are

• GitHub data Extractor
• Hivemind Discord
• Hivemind Discourse
• Hivemind GitHub
• Hivemind Telegram
• Hivemind Google-Drive
• Hivemind MediaWiki
• Hivemind Notion
• Violation Detection
• Telegram Analyzer
• All Discord Guilds Analyzer
• Single Discord Guild Analyzer
• Discourse Analyzer

Notes:

• The Hivemind DAGs handle data ingestion for TogetherCrew's RAG (Retrieval-Augmented Generation) pipeline.
• The Analyzer DAGs processes platform data through TogetherCrew's general analytics engine.

Running the App

Follow these steps to run the Airflow DAGs:

1. Set up environment files:

   • Copy .env.airflow.init.sample to .env.airflow.init
   • Copy .env.airflow.sample to .env.airflow
   • Configure your credentials in both files

2. Start Airflow using Docker Compose:

   docker-compose -f docker-compose.yaml up

reference: Running Airflow in Docker