Home

Intro

This code hits several endpoints based on clients built to the API specs, grooms and deidentifies data to city spec, and uploads the groomed data into prepared Socrata endpoints.

This is the companion repo to the stat_dashboard_frontend

Detailed Docs

Architecture

Issues

Please file found bugs or other issues here in the github repo, it's maintained and monitored.
Would you like to contribute a fix? Please do!

Install

git clone https://github.com/cityofsomerville/stat_dashboard_pipeline.git
cd stat_dashboard_pipeline

Development

pip install -e .

Production

pip install .

Credentials

Copy the config/sample_auth.yaml file and rename it to auth.yaml.
Put the credentials in the file within the empty quotation marks and save. Google Analytics credentials are pulled from Google Analytics via a JSON file. NOTE: To run the QScend API, you'll need to get your endpoint IP address passlisted by QScend support.

Run

stat_pipeline

Initialize

NOTE: Does not store in Socrata
To initialize the data for the datasets: Simple one-off queries that dump to CSV files for upload to the Somerville Socrata instance via the UI.

stat_pipeline -i

Migrate

Run a complete historical migration of QScend activities (takes some time)

stat_pipeline -m

Run Unit Tests

NOTE: These should run as acceptance tests in github

python setup.py test

Lint