TPC - DI - Benchmark artefacts for Cloud Composer and BigQuery

Install Airflow:

Create Airflow Docker Image

1. Clone Repo Airflow Docker
2. Build a docker image from

   cd docker-airflow
   docker build --rm --build-arg AIRFLOW_DEPS="gcp" -t tpc-di/benchmark-airflow .

3. Ensure image is created by executing

   docker images

Start Airflow

1. Run the following command to start Apache Airflow

   docker-compose up -d

2. Navigate to Airflow Web UI

Create a Service Account in GCP for Airflow to use

1. Navigate to GCP BigQuery API and enable it if not already enabled
2. Navigate to the Credential Page
3. Click on Create Credential button and choose Service Account
4. Fill in Service Account Name and ID
5. Click on Create
6. Grant BigQuery Admin and Storage Admin roles and Continue
7. Click on Create Key and choose JSON key
8. Download generated JSON key and keep it safe

Create connections in Airflow Admin UI

1. Navigate to Admin -> Connections Connections Page
2. Find bigquery_default connection or equivalent and edit to get the edit window
3. Fill in the contents of keyfile.json from GCP obtained in earlier step and save connection
4. Repeat the above steps to setup connection for google_cloud_default