Skip to content

Latest commit

 

History

History
85 lines (66 loc) · 2.65 KB

README.md

File metadata and controls

85 lines (66 loc) · 2.65 KB
Raw

2022 Code

Burma Price Monitor

  • Built with -> requests-html , pandas , datasette , github-action, cloud-run, bigquery, sqlite, playwright

About

The base idea of this project is to record the daily struggles of Burma People to be remembered in the furture and the significant point to monitor is price of commodity or groceries.

Current tracking prices - Rice , Common groceries , Petrol

Visualization

Data Sources

  1. Petrol price from Myanmar Petroleum Trade Association (short : MPTA)
  2. Common gorceries price from Wisarra
  3. Rice data from Myanmar Rice Association Facebook Page. (Not update yet)
  4. Daily price rates from Denko company website. (starting from 2022 May 19)
  5. Startfish Petroleum port price. (starting from 2019 Dec 29 to current day)

Installation

  • Install required libraries :
pip install -r requirements.txt
  • Run scraping task - code
python3 scrape_marketprice.py
  • Deploy to datasette & cloud run
./deploy.sh

Workflow

  1. Daily collect/scrape data from websites.
  2. Automate with Github Action.
  3. Deploy with Datasette to Google Cloud run service.

Remaining to do 📝

  • Data Viz

    • Create Overall dashboard
    • Embed dashboard into webpage

  • Datasette

    • Manual in Burmese
    • Data cleaning script
      • Make sure to use correct data formats in important columns
      • Should I add another .db for clean data?
    • Relational Database between different data sources
    • Add custom template for UI
    • Update metadata.yml

  • Myanmar Rice

    • Collect data from Gsheet into BigQuery
    • BigQuery into SQlite db with python
    • Insert into price.db

  • MPTA :

    • Update to Datasette
    • Automate with Github Action

  • Wisarra

    • Update to Datasette

  • Added resources

    • GreenWay Market Price
    • MaxMyanmar
      • Automate with Github Action
      • Scrape data - Finished at June 3,22. It took me two or three days to finish it and the most difficult task before I found out how to call POST method JSON. HaHa , Good to learn new things. First time using Playwright.
    • Denko - May 19
    • Starfish Petrol - May 21

Mini_Project

  • Scraping Tableau Featured Developer page
    • Code
    • Main idea is to make monthly scraper to collect Tableau Dev data and make an analysis/viz.
    • Buit with -> Selenium , BeautifulSoup , Datasette , Python