🧪 pap-datalab

Data laboratory for Punan ang Patlang project.

📦 Dependencies

Using pap-datalab requires

• up-and-running instance of pap-orchestrator compose stack
• installation of poetry
• .env file containing the following (this depends on your instance of pap-orchestrator)
  • clickhouse_username
  • clickhouse_password

✅ Instructions

Database Structure

1. Under the pap-orchestrator's compose stack, navigate to pap-clickhouse
2. In pap-clickhouse's Web SQL UI (by default http://localhost:18123/), run all the SQL files under sql/database/. This will create databases separated by data source
3. Also in pap-clickhouse's Web SQL UI, run all SQL files under sql/table/. This will create the initial set of tables with the specified schema under each database (but no data yet!)

Populating the Database

1. Run poetry install in the repository's root folder to install all dependencies
2. Run all notebooks under notebooks/

📍 What's next

You should now have a sort of working database stack + data. You can now

• Use the Apache Superset Instance to:
  • Explore your data (SQL Lab);
  • Create charts (Charts);
  • or create dashboards (Dashboard)
• Create new data generating notebooks