Make a rcp2 package proof of concept.

[nbanion]

Purpose

It should be easy for volunteers and Red Cross partners to use the rcp2 project. By packaging our code and investing in project infrastructure, we can make it easier to maintain, document, and reproduce our work.

Objectives

1. Discuss our needs, guiding principles, and vision for the high level infrastructure.
2. Implement an extensible proof of concept (POC) for the desired infrastructure.

The POC won't be comprehensive. We can close this issue after, say, setting up the infrastructure for one data source. After that, we can add other issues to build out the package infrastructure.

[nbanion]

Some opening thoughts.

@kelsonSS, @thwhitfield, @jessezlotoff, i welcome your thoughts. I'd like to hear from Ian and Jake too; I just couldn't find their handles right away. Let me know if you'd add, remove, or modify any points.

Audience

As I understand, there are two audiences for this package.

• Volunteers want to set up quickly and leverage existing work with confidence.
• Red Cross Partners want to generate insights and update data sources with minimal fiddling.

Needs

The package should make it easy to:

• Download raw data from original sources.
• Clean raw data so that it's ready to use.
• Extract features that are commonly used for modeling.
• Predict results using models we've trained.
• Summarize results in maps, tables, reports, etc.

I could imagine one or more modules for each of these different domains.

Guiding Principles

• Encourage reproducibility by automating all data handling steps from the untouched raw data.
• Document package components so that our work is easy for others to use and maintain.
• Use unit tests to make sure individual parts work.
• Keep it simple, and don't design for features you don't yet need.
• Use notebooks for prototyping, but not for infrastructure.

[nbanion]

For implementation, I propose building on the the Cookiecutter Data Science template we're already using.

rcp2/
  setup.py        # Set up package infrastructure.
  Makefile        # Coordinate data processing steps.
  data/
    raw/          # Untouched raw data from download.
    processed/    # Cleaned data for modeling.
  src/
    __init__.py   # Initialize the package.
    utils.py      # General purpose utilities (e.g., directory navigation).
    download.py   # Utilities to download data.
    clean.py      # Utilities to clean raw data.
    features.py   # Utilities for extracting features.
    model.py      # Utilities for using models.
    summarize.py  # Utilities for summarizing results.
    tests/
      test_*.py   # Unit tests.
  models/         # Trained and serialized models.
  docs/           # Sphinx documentation for package.

The package guts are in src/. I could imagine implementing utils.py, download.py, and clean.py for this POC.

This outline persists both the raw data and the processed data locally. The data I've seen so far only amount to a couple GB, so I don't see a huge space issue with keeping them locally. I'd be open to limiting the processed/ dir to just those datasets that process slowly and benefit from being cached. Everything else could be processed on the fly from package code.

I recommend using a Makefile to coordinate data processing steps. The make command intelligently runs scripts in the order of their dependencies, and it skips running scripts whose output files are already up to date. Looks like it's possible to use make or something like it on linux, Mac, and Windows-- though a little more research may be required. That said, using make results in clean, concise commands for reproducing an analysis.

$ make download  # Possible command to do all the downloading.

I envision that a lot of volunteer work would continue to happen in notebooks/. Volunteers could import src at the top of their notebooks to leverage existing data processing. And as they refined their work, they could migrate it to src so that others can easily use it too.

Last, we could build out the existing Sphinx documentation and maybe post it to Read the Docs.

[thwhitfield]

I wholeheartedly agree with this approach, and think the framework that you've laid out is great.

I've been working with Jake to develop the code for importing and doing initial cleaning of the raw NFIRS data, so we can start with that if we'd like. The only caveat is that Jake gets those files on a CD and has to do a manual export of the files on the CD to csv files. But from then on I think we can write scripts and a makefile to automate the rest of the process without too much trouble.

[nbanion]

Sounds like a great starting point.

I think we could just treat Jake's CSVs on Google Drive as the raw input for NFIRS. Looks like you may be leaning that way too on #61 .

[nbanion]

Insight from today's meeting: make sure the final product is easy for our audience to use.

Once this work is a bit more developed, I think it would be great to test drive with Jake to make sure all goes well in his environment.

Will also plan to keep the logic in unit-tested python code, and to keep the sequence of scripts simple.

[kelsonSS]

@nbanion Thanks for keeping this issue so well documented!

As we discussed my feelings should be that we have most of the heavy lifting done in one python script file that will sequentially call the other files. this file can then be packaged into the makefile for convenience.

That way if for whatever reason the makefile doesn't work the user can just use the python main file as a simple alternative

[ChiefFireDataNerd]

As I noted on the call. The approach you've outlined is very well documented and makes a whole lot of sense...

I did have a question about the "audience" you identified at the top. Are you suggesting that John Q ARC Volunteer is going to use the code or notebook themselves? I had assumed that the end users would only be using some sort of visualization tool a bit like the original output: https://home-fire-risk.github.io/smoke_alarm_map/

Also it seems unlikely that the people asked to support this (in the event that Jake is unable to get it working himself, gets a huge promotion and is too busy to work on things like this, or (excuse the phrase) is hit by a bus, metaphorically speaking) would be ideally suited to support this sort of approach. We should make sure that this tool is easy for the every day IT person to figure out how to trouble shoot or do minor bug fixes. And I'm not convinced that most run-of-the-mill IT people are adept at python, make, or the like.

[nbanion]

@ChiefFireDataNerd , thanks-- I agree that priority 1 is knowing our audience.

I think this approach does serve DataKind volunteers like us by making it easy to organize and replicate our work. On the call, folks mentioned that data management challenges are growing along with modeling work. That audience alone might be the motivator for this approach.

I don't know the Red Cross audience as well. Questions for the group:

• What does the client need to be able to do with this project?
• What kind of interface would the client be comfortable with?
• Do we plan to provide ongoing maintenance after hand-off?
  • Address bugs reported on GitHub?
  • Help with adding new data?
  • Answer client questions?

The visualization tool seems great for insights, and I could imagine building additional features that allow users to manage data using a graphical user interface. That said, I assume that those features would rely in part on the package that we're discussing here, and that bug fixing will inevitably involve working with source code.

Perhaps the Red Cross user interface merits an issue of its own?