Perf: At-risk dashboard issues

[bmtcril]

Tested with the 300M row dataset on ClickHouse Cloud I've found the following issues:

OOM:

• Watched Video Segments (at-risk)
• Problem Interactions (at-risk)
• Partial and Full Views Per Video (at-risk)
• Problem Results (at-risk)

Slow queries (> 3 seconds):

1. learner-groups - Problems attempted per section/subsection (at-risk)
   Superset time: 20.888254 (s).
   Query duration: 20.439 (s).
   Result rows: 44
   Memory Usage (MB): 4332.254327774048
   Row count (superset) 44
   Filters: [{'column': 'course_key'}]

2. learner-groups - Video Views per Section/Subsection (at-risk)
   Superset time: 11.049571 (s).
   Query duration: 10.62 (s).
   Result rows: 25
   Memory Usage (MB): 3735.4167289733887
   Row count (superset) 25
   Filters: [{'column': 'course_key'}]

3. learner-groups - Page views per section/subsection (at-risk)
   Superset time: 10.707674 (s).
   Query duration: 10.328 (s).
   Result rows: 37
   Memory Usage (MB): 3852.6872720718384
   Row count (superset) 37
   Filters: [{'column': 'course_key'}]

4. learner-groups - Cumulative Interactions (at-risk)
   Superset time: 10.587901 (s).
   Query duration: 8.761 (s).
   Result rows: 5
   Memory Usage (MB): 2674.5562772750854
   Row count (superset) 5
   Filters: [{'column': 'visited_on'}, {'column': 'course_key'}]

5. learner-groups - Subsection Summary (at-risk)
   Superset time: 10.024011 (s).
   Query duration: 9.625 (s).
   Result rows: 30
   Memory Usage (MB): 2833.420473098755
   Row count (superset) 30
   Filters: [{'column': 'visited_on'}, {'column': 'course_key'}]

6. learner-groups - Section Summary (at-risk)
   Superset time: 9.862307 (s).
   Query duration: 9.21 (s).
   Result rows: 7
   Memory Usage (MB): 2681.8192348480225
   Row count (superset) 7
   Filters: [{'column': 'visited_on'}, {'column': 'course_key'}]

7. learner-groups - Video Views by Section/Subsection (at-risk)
   Superset time: 9.843626 (s).
   Query duration: 9.401 (s).
   Result rows: 40
   Memory Usage (MB): 2461.7836952209473
   Row count (superset) 40
   Filters: [{'column': 'emission_time'}, {'column': 'course_key'}]

8. learner-groups - Evolution of Engagement (at-risk)
   Superset time: 9.206902 (s).
   Query duration: 8.761 (s).
   Result rows: 5
   Memory Usage (MB): 2674.5562772750854
   Row count (superset) 5
   Filters: [{'column': 'visited_on'}, {'column': 'course_key'}]

9. learner-groups - Last visit date
   Superset time: 6.424135 (s).
   Query duration: 5.688 (s).
   Result rows: 8114
   Memory Usage (MB): 1887.9568014144897
   Row count (superset) 8114
   Filters: [{'column': 'course_key'}]

10. learner-groups - At-risk learners
    Superset time: 5.895613 (s).
    Query duration: 5.317 (s).
    Result rows: 1000
    Memory Usage (MB): 1881.2105741500854
    Row count (superset) 1000
    Filters: [{'column': 'course_key'}]

11. learner-groups - At-risk Enrollment Dates
    Superset time: 5.881555 (s).
    Query duration: 5.076 (s).
    Result rows: 8119
    Memory Usage (MB): 1886.271502494812
    Row count (superset) 8119
    Filters: [{'column': 'course_key'}]

12. learner-groups - At-risk Enrollees per Enrollment Track
    Superset time: 5.200881 (s).
    Query duration: 4.766 (s).
    Result rows: 3
    Memory Usage (MB): 1881.2149724960327
    Row count (superset) 3
    Filters: [{'column': 'course_key'}]

13. learner-groups - Distribution of Course Grades
    Superset time: 5.158168 (s).
    Query duration: 4.801 (s).
    Result rows: 10
    Memory Usage (MB): 1883.7704000473022
    Row count (superset) 10
    Filters: [{'column': 'course_key'}]

[bmtcril]

There should be decent improvements in these after #822 and the work on problems in dbt.

[crathbun428]

@bmtcril - I see the linked issue was closed... can we mark this as done or should we keep it as doing?

[bmtcril]

I think we can close these, we need to do another round of testing now that we have finalized the dashboards and can create new issues if need be.