Issue 917: Create a markdown of the synthetic recover results & post …

…as a PR comment (#928)

* add .md file

* add figs not on GH as a test, run eval in actions

* update to include infection figures

* remove old commits of figures

* add crete.dir to plotting function so saves if figs not committed

* add run of eval_rt.R

* Attempt to render and post as a comment the markdown

* fix typo

* remove slash

* rmarkdown as pkg

* test removing other R script

* copy render readme package install

* setup panic

* run both scripts

* remove eval_rt to test

* remove .md dependency on eval_rt

* try specifying artifact in posting comment

* try a different PR artifact

* add prefix

* typo

* try live-codes

* remove message

* add back eval_rt with extra packages

* add back in eval rt figs to md

* use live-codes for previewing artifact

* create new live codes action to comment

* add .livecodes directory

* try out post-artifact action

* delete livecodes

* remove extra md

* fix typos

* upload rendered markdown

---------

Co-authored-by: Kaitlyn Johnson <kaitlyn.johnson@lshtm.ac.uk>

.github/workflows/synthetic-validation.yaml

@@ -13,14 +13,18 @@ on:
   workflow_dispatch:
 
 jobs:
-  synthetic-valiation:
+  synthetic-validation:
     runs-on: macos-latest
     env:
       GITHUB_PAT: ${{ secrets.GITHUB_TOKEN }}
     steps:
       - uses: actions/checkout@master
 
-      - uses: r-lib/actions/setup-r@v2
+      - name: Setup R
+        uses: r-lib/actions/setup-r@v2
+
+      - name: Setup pandoc
+        uses: r-lib/actions/setup-pandoc@v2
 
       - uses: r-lib/actions/setup-r-dependencies@v2
         with:
@@ -30,12 +34,17 @@ jobs:
             here
             scoringutils
             loo
+            data.table
+            rstan
+            any::rmarkdown
             matrixStats
             local::.
       - name: Run synthetic validation
         run: |
-          source("inst/dev/recover-synthetic/rt.R")
-        shell: Rscript {0}
+          Rscript -e '
+          source("inst/dev/recover-synthetic/rt.R") 
+          source("inst/dev/recover-synthetic/eval_rt.R") 
+          '
 
       - name: Upload validation figures
         uses: actions/upload-artifact@v4
@@ -50,3 +59,29 @@ jobs:
           name: fits
           retention-days: 5
           path: synthetic.rds
+
+      - name: Render synthetic recovery md
+        run: |
+          rmarkdown::render("inst/dev/synthetic_recovery.md")
+        shell: Rscript {0}
+
+      - name: Upload markdown
+        uses: actions/upload-artifact@v4
+        with:
+          name: synthetic_recovery
+          retention-days: 5
+          path: inst/dev/synthetic_recovery.html
+
+      - name: Post the artifact
+        uses: CDCgov/cfa-actions/post-artifact@main
+        if: ${{ github.event_name == 'pull_request' }}
+        with:
+          artifact-name: synthetic_recovery
+          gh-token: ${{ secrets.GITHUB_TOKEN }}
+          message: 'Thank you for your contribution ${{ github.actor }} :rocket:! Your { artifact-name } markdown is ready for download :point_right: [here]({ artifact-url }) :point_left:!'
+
+
+
+
+
+

inst/dev/recover-synthetic/plot.R

@@ -17,7 +17,8 @@ save_ggplot <- function(plot, name, prefix = "") {
   ggplot2::ggsave(
     here::here("inst", "dev", "figs", paste0(prefix, name, ".png")),
     plot,
-    dpi = 300, width = 9, height = 6
+    dpi = 300, width = 9, height = 6,
+    create.dir = TRUE
   )
 }
 

inst/dev/synthetic_recovery.md

@@ -0,0 +1,32 @@
+# Synthetic Recovery visual results
+
+This file combines the figures generated in `rt.R` into a single file, used to 
+confirm that the fits to synthetic data with different EpiNow2
+specifications work as expected. 
+
+In all figures, the black dots indicate the ground truth R(t) estimate, used to 
+generate the synthetic data used for model fitting. 
+
+### Estimate from the default Gaussian Process settings using the No-U-Turn (NUTS) sampler
+![](./figs/rt_gp_nuts.png)
+![](./figs/inf_gp_nuts.png)
+
+### Estimate from back-calculation
+![](./figs/rt_backcalc_nuts.png)
+![](./figs/inf_backcalc_nuts.png)
+
+### Estimate from a weekly random walk, with no Gaussian process
+![](./figs/rt_weekly_rw_nuts.png)
+![](./figs/inf_weekly_rw_nuts.png)
+
+### Estimate from a monthly random walk + a stationary Gaussian process
+![](./figs/rt_gp_rw_nuts.png)
+![](./figs/inf_gp_rw_nuts.png)
+
+
+### Comparison of different model specs
+CRPS over time evaluated against known R(t)
+![](./figs/rt_crps.png)
+CRPS over time evaluated against infections
+![](./figs/inf_crps.png)
+