Skip to content

Commit

Permalink
Update README.md
Browse files Browse the repository at this point in the history
  • Loading branch information
Pce-iyos authored Jun 20, 2024
1 parent dd6ae89 commit d78980c
Showing 1 changed file with 53 additions and 0 deletions.
53 changes: 53 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,12 @@ SpicyBytes is an innovative food and grocery management platform aimed at reduci
├───.github
│ └───workflows
├───dags
│ ├───allminogcs.py
│ ├───collector.py
│ ├───etl_exploitation_zone.py
│ ├───etl_formatted_zone.py
│ ├───expiry_notification.py
│ └───synthetic.py
├───data
│ └───raw
├───landing_zone
Expand All @@ -33,6 +39,32 @@ SpicyBytes is an innovative food and grocery management platform aimed at reduci
│ ├───customer_purchase
│ ├───sentiment_reviews
│ └───supermarket_products
├───formatted_zone
│ ├───business_review_sentiment.py
│ ├───customer_location.py
│ ├───customer_purchase.py
│ ├───customer_sales.py
│ ├───customers.py
│ ├───dynamic_pricing.py
│ ├───establishments_catalonia.py
│ ├───estimate_expiry_date.py
│ ├───estimate_perishability.py
│ ├───expiry_notification.py
│ ├───individual_review_sentiment.py
│ ├───location.py
│ └───mealdrecomend.py
├───exploitation_zone
│ ├───dim_cust_location.py
│ ├───dim_date.py
│ ├───dim_product.py
│ ├───dim_sp_location.py
│ ├───fact_business_cust_purchase.py
│ ├───fact_business_inventory.py
│ ├───fact_business_review.py
│ ├───fact_cust_inventory.py
│ ├───fact_cust_purchase.py
│ ├───fact_customer_review.py
│ └───schema.txt
└───readme_info
```

Expand Down Expand Up @@ -67,3 +99,24 @@ The `collector.py` DAG collects data on a monthly basis, while the `synthetic.py
The proposed high level architecture is employed for the P1 delivery methodology.


# SpicyBytes P2 Delivery : Formatted Zone and Exploitation Zone

## DAGs

We have created several DAGs to manage the workflows within the following zones:

1. **Formatted Zone**
- Manages the tasks related to data formatting and standardization.
- Sends formatted files to Google Cloud Storage.
2. **Exploitation Zone**
- Handles data exploitation, including analysis and transformation tasks.
- Sends data to BigQuery and connects to Google Looker for further analysis and visualization.
3. **Landing Zone**
- Manages the initial data landing, ingestion, and raw data handling.

## How to Use

1. **Setting Up**: Ensure all dependencies are installed and the environment is configured properly.
2. **Executing DAGs**: The DAGs can be executed via the Airflow scheduler. Ensure the Airflow server is running and the DAGs are enabled in the Airflow UI.
3. **Monitoring**: Monitor the execution of the DAGs through the Airflow UI for any errors or required interventions.

0 comments on commit d78980c

Please sign in to comment.