RD 10/22/14

README.md

@@ -2,22 +2,28 @@
 
 This repository has two main components:
 
-1.  Indicator framework for quickly generating aggregate indicators.
-2.  Reporting framework for generating reports from those indicators.
+1.  Aggregate table framework for generating aggregate tables.
+2.  Reporting framework for generating reports from those aggregate tables.
 
-## Indicator framework
+## Aggregate table framework
 
-To build the monthly and lifetime indicator tables issue the following
-command:
+To build the aggregate tables issue the following
+commands:
 
-    make indicators
+    make clean
+    make aggregate_tables
 
-This calls [indicators.R](indicators.R), which creates
-tables described in [indicators.json](indicators.json)
-and [indicator_functions.R](indicator_functions.R).
+Alternatively, make debug can be used to build aggregate tables from a subset of rows from each data source. This is useful for data sources that involve long-running queries. Specify debug mode as follows:
+
+    make clean
+    make debug
+
+Both forms call [aggregate_tables.R](aggregate_tables.R), which creates
+tables described in [aggregate_tables.json](aggregate_tables.json)
+using the indicator calculation functions in [indicator_functions.R](indicator_functions.R).
 
 How does one add an indicator?
-[indicators.json](indicators.json) contains a list of
+[aggregate_tables.json](aggregate_tables.json) contains a list of
 tables to be created. Each element in the list is a dictionary with
 four pairs:
 
@@ -28,31 +34,26 @@ four pairs:
 4.  A list of components that will be merged together to build the
     final table.
 
-Each component has three key-value pairs:
+Each component has two key-value pairs:
 
-1.  The name of the database where the data for this component is
-    pulled.
-2.  The name of the table where the data for this component is pulled.
-3.  The columns to be created.
+1.  The name of the table where the data for this component is pulled.
+2.  The columns to be created.
 
 Here is an example configuration:
 
     [
         {
-            "database": "dimagi_data_platform",
-            "table": "monthly",
+            "table": "aggregate_monthly_interactions",
             "by": ["domain", "user_id", "month.index"],
             "components": [
                 {
-                    "database": "dimagi_data_platform",
                     "table": "interactions",
                     "columns": {
                         "column1": "date_first_visit",
                         "column2": "date_last_visit"
                     }
                 },
                 {
-                    "database": "dimagi_data_platform",
                     "table": "device_logs",
                     "columns": {
                         "column3": "nrow"
@@ -62,12 +63,11 @@ Here is an example configuration:
         }
     ]
 
-This will create a new table called `monthly` in the
-`dimagi_data_platform` with three columns. `column1` will be the date
-of the first visit a user performed in the month. `column2` will be
-the date of the last visit a user performed in the month. `column3`
-will count the number of device logs recorded in a month for a given
-user. Notice that the key for each column is the name of the column
+This will create a new table called `aggregate_monthly_interactions` with three columns. 
+`column1` will be the date of the first visit a user performed in the month. 
+`column2` will be the date of the last visit a user performed in the month. 
+`column3` will count the number of device logs recorded in a month for a given user. 
+Notice that the key for each column is the name of the column
 where that indicator will be stored, the value for each column is the
 R function to be called on each block of data. To modularize the code
 a bit, we have moved custom indicator functions into
@@ -78,9 +78,18 @@ the definition of `date_first_visit`:
 
 This function takes a data.frame `x` and returns the minimum value of
 the `visit_date` column of `x`. So, adding an indicator always
-requires modifying [indicators.json](indicators.json)
+requires modifying [aggregate_tables.json](aggregate_tables.json)
 and if the necessary R function to calculate the indicator does not
 exist then it must be added to
 [indicator_functions.R](indicator_functions.R).
 
 ## Reporting framework
+Report modules produce one or more PDF reports. Report modules for a run are specified in config_run.json, along with options like split-bys and report dates. 
+
+test_report.R is an example report module.
+
+The script reports.R reads the config file, sets up variables (including filtering domains and returning a list of domains to include in the report run) and runs the report modules specified in the config.  Each report module should return a list of file names (full paths) specifying the individual pdfs it has written to tmp_pdf_report_dir. Once all report modules have run, these are collated into a single pdf in reports.R, using http://www.pdflabs.com/tools/pdftk-the-pdf-toolkit/.
+
+Report modules should provide a render function that returns a list of pdf files (full paths) generated in the report module. The render function should be defined as follows:
+
+    render <- function (db,domains_for_run,report_options,tmp_report_pdf_dir)

aggregate_tables.R

@@ -29,26 +29,28 @@ drop_tables <- function(file) {
 
 write_tables <- function(file, debug) {
     config <- fromJSON(file=file)
-
+    db <- get_db_connection()
     for (table.info in config) {
-        print(paste('Writing', table.info$table, 'indicator table.'))
-        df <- compute_indicators(table.info, debug)
-        db <- get_db_connection()
+        print(paste('Computing indicators for ', table.info$table, 'indicator table.'))
+        df <- compute_indicators(table.info, db, debug)
+        print(paste('Writing ', table.info$table, 'indicator table.'))
         dbRemoveTable(db$con, name=table.info$table)
         copy_to(db, df=df, name=table.info$table, temporary=FALSE)
     }
 }
 
-compute_indicators <- function(info, debug) {
+compute_indicators <- function(info, db, debug) {
   debug <- as.logical(debug)
   if (debug == T) {limit = 5000} else {limit = -1}
     dfs <- lapply(info$components, function(component) {
-        db <- get_db_connection()
+        print(paste('Getting data source ', component$table))
         source.data <- get_data_source(db, component$table, limit)
         group.by.str <- paste(info$by, collapse=', ')
+        print(paste('Grouping and aggregating', component$table))
         df <- source.data %.% s_group_by(group.by.str) %.% aggregate(component$columns)
         return(df)
     })
+    print('merging...')
     merged <- Reduce(function(...) merge(..., all.x=TRUE, all.y=TRUE, by=info$by), dfs)
     return(merged)
 }

aggregate_tables.json

@@ -3,41 +3,58 @@
       "table":"aggregate_lifetime_interactions",
       "by":[
          "domain",
-         "user_id"
+         "user_id",
+         "user_pk"
       ],
       "components":[
          {
-            "table":"interactions",
+            "table":"visit_detail",
             "columns":{
                "date_first_visit":"date_first_visit",
                "date_last_visit":"date_last_visit",
                "nvisits":"nvisits",
-               "days_on_cc":"days_on_cc",
-               "days_visit_last":"days_visit_last",
-               "active_user":"active_user",
                "calendar_month_on_cc":"calendar_month_on_cc",
-               "active_months":"active_months",
-               "active_month_percent":"active_month_percent",
-               "active_days":"active_days",
-               "active_day_percent":"active_day_percent",
                "nforms":"nforms",
                "median_visit_duration":"median_visit_duration",
                "median_visits_per_day":"median_visits_per_day",
                "median_visits_per_month":"median_visits_per_month",
-               "median_time_elapsed_btw_visits":"median_time_elapsed_btw_visits",
-               "batch_entry_visit":"batch_entry_visit",
-               "batch_entry_percent":"batch_entry_percent",
-               "ncases_registered":"ncases_registered",
-               "register_followup":"register_followup",
-               "case_register_followup_rate":"case_register_followup_rate",
                "morning":"morning",
                "afternoon":"afternoon",
+               "evening":"evening",
                "night":"night",
-               "after_midnight":"after_midnight",
-               "ncases_opened":"ncases_opened",
+               "time_using_cc":"time_using_cc",
+               "nvisits_travel":"nvisits_travel",
+               "nvisits_travel_batch":"nvisits_travel_batch"
+            }
+         },
+         {
+            "table":"interactions",
+            "columns":{
+               "ninteractions":"ninteractions",
+               "ncases_registered":"ncases_registered",
+               "register_followup":"register_followup",
+               "case_register_followup_rate":"case_register_followup_rate",
                "ncases_touched":"ncases_touched",
                "nunique_followups":"nunique_followups"
             }
+         },
+         {
+            "table":"device_type",
+            "columns":{
+               "summary_device_type":"summary_device_type",
+               "nforms": "nforms",
+               "active_days":"active_days",
+               "active_months":"active_months"
+
+            }
+         },
+         {
+            "table":"device_log_types_by_user",
+            "columns":{
+               "total_logs":"total_logs",
+               "audio_plays":"audio_plays",
+               "network_warnings":"network_warnings"
+            }
          }
       ]
    },
@@ -46,40 +63,53 @@
       "by":[
          "domain",
          "user_id",
+         "user_pk",
          "month.index"
       ],
       "components":[
          {
-            "table":"interactions",
+            "table":"visit_detail",
             "columns":{
                "date_first_visit":"date_first_visit",
                "date_last_visit":"date_last_visit",
                "nvisits":"nvisits",
-               "days_on_cc":"days_on_cc",
-               "active_days":"active_days",
-               "active_day_percent":"active_day_percent",
-               "nforms":"nforms",
                "median_visit_duration":"median_visit_duration",
                "median_visits_per_day":"median_visits_per_day",
-               "median_time_elapsed_btw_visits":"median_time_elapsed_btw_visits",
-               "batch_entry_visit":"batch_entry_visit",
-               "batch_entry_percent":"batch_entry_percent",
-               "ncases_registered":"ncases_registered",
-               "register_followup":"register_followup",
-               "case_register_followup_rate":"case_register_followup_rate",
                "morning":"morning",
                "afternoon":"afternoon",
+               "evening":"evening",
                "night":"night",
-               "after_midnight":"after_midnight",
-               "ncases_opened":"ncases_opened",
+               "numeric_index":"numeric_index",
+               "time_using_cc":"time_using_cc",
+               "nvisits_travel":"nvisits_travel",
+               "nvisits_travel_batch":"nvisits_travel_batch"
+            }
+         },
+         {
+            "table":"interactions",
+            "columns":{
+               "ninteractions":"ninteractions",
+               "ncases_registered":"ncases_registered",
+               "register_followup":"register_followup",
+               "case_register_followup_rate":"case_register_followup_rate",
                "ncases_touched":"ncases_touched",
                "nunique_followups":"nunique_followups"
             }
          },
          {
             "table":"device_type",
             "columns":{
-               "summary_device_type":"summary_device_type"
+               "summary_device_type":"summary_device_type",
+               "nforms": "nforms",
+               "active_days":"active_days"
+            }
+         },
+         {
+            "table":"device_log_types_by_user",
+            "columns":{
+               "total_logs":"total_logs",
+               "audio_plays":"audio_plays",
+               "network_warnings":"network_warnings"
             }
          }
       ]