Feature creation

[dominicwhite]

To-do

• Merge all the individual datasets (in the data/raw and data/static folders) into a joined table.

  • For block level data (partially addressed by pull request Data processing and dashboard prototype #6 )

  • For tract level data

• Create any new features from raw census/ACS data that require calculation before being displayed in the final dataset :

  • The dashboard will show data about (1) the nationality, (2) the ethnicity, and the (3) languages that are present in each block group (and tract). For example, when a user hovers over a polygon, 3 pie charts might appear, showing the composition of each of these variables in that area.

  • We should pre-create these variables during the data pipeline stage, so that the Shiny app can just render a column in the dataset, rather than having to do a bunch of calculations that might increase latency.

  • The (currently anticipated) variables that we need are:

    • The % of each nationality in each block group/tract.

    • The % of each language in each block group/tract.

    • The % of each ethnicity in each block group/tract

[dominicwhite]

We should better document the variables that are in the downloaded data, and the ones in the final, transformed data.

We also need to decide on human readable labels to display in the dashboard (rather than the current mangled/inconsistent column names).