[RW-12298] upgrade AoU docker images

[yonghaoy]

Jira ticket: https://broadworkbench.atlassian.net/browse/[ticket_number]

Summary of changes

What

Why

Testing these changes

What to test

Who tested and where

• This change is covered by automated tests
  • NB: Rerun automation tests on this PR by commenting jenkins retest or jenkins multi-test.
• I validated this change
• Primary reviewer validated this change
• I validated this change in the dev environment

[Qi77Qi]

which version we have in prod today? this needs to be what's in prod

[yonghaoy]

good point. it is 2.2.7 now.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 73.64%. Comparing base (2f03d6c) to head (a1c4e02).
Report is 2 commits behind head on develop.

Additional details and impacted files

@@           Coverage Diff            @@
##           develop    #4490   +/-   ##
========================================
  Coverage    73.64%   73.64%           
========================================
  Files          158      158           
  Lines        14694    14694           
  Branches      1162     1162           
========================================
  Hits         10822    10822           
  Misses        3872     3872           

---

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2f03d6c...a1c4e02. Read the comment docs.

[Qi77Qi]

seems dataproc was updated at some point https://github.com/DataBiosphere/leonardo/blob/develop/jenkins/dataproc-custom-images/prepare-custom-leonardo-jupyter-dataproc-image.sh#L24

[yonghaoy]

seems dataproc was updated at some point https://github.com/DataBiosphere/leonardo/blob/develop/jenkins/dataproc-custom-images/prepare-custom-leonardo-jupyter-dataproc-image.sh#L24

The latest image on GCR is 2.2.9 https://console.cloud.google.com/artifacts/docker/broad-dsp-gcr-public/us/us.gcr.io/terra-jupyter-aou?project=broad-dsp-gcr-public
But the version should be 2.2.11 actually: https://github.com/DataBiosphere/terra-docker/pull/485/files
There might be some process is broken, not sure what happened

[Qi77Qi]

java.lang.RuntimeException: google project could not be claimed for test, java.util.concurrent.TimeoutException: Future timed out after [5 minutes]

just triggered rerun of all tests

[yonghaoy]

java.lang.RuntimeException: google project could not be claimed for test, java.util.concurrent.TimeoutException: Future timed out after [5 minutes]

just triggered rerun of all tests

java.lang.RuntimeException: google project could not be claimed for test, java.util.concurrent.TimeoutException: Future timed out after [5 minutes]
I don't think that is related

[yonghaoy]

java.lang.RuntimeException: google project could not be claimed for test, java.util.concurrent.TimeoutException: Future timed out after [5 minutes]

just triggered rerun of all tests

java.lang.RuntimeException: google project could not be claimed for test, java.util.concurrent.TimeoutException: Future timed out after [5 minutes] I don't think that is related

DataBiosphere/terra-resource-buffer#294

[Qi77Qi]

I have a feeling we might run out of disk...but can verify Mon on a BEE

[yonghaoy]

I have a feeling we might run out of disk...but can verify Mon on a BEE
Having issues with bee, forget about how to use it.

For image size, seems not a big increase

• gce: 42GB to 44GB
• datapro 47 -> 47

[LizBaldo]

FYI @yonghaoy and @Qi77Qi I am taking a stab at updating the docker images again with a new bioconductor release and I will try to debug this while I am at it, I will keep you posted DataBiosphere/terra-docker#488

[Qi77Qi]

thanks @LizBaldo !
haven't looked recently, but last time I looked, sth is preventing runtime from restarting

[LizBaldo]

Got it, the only thing that I remember doing back in March is fixing the AOU image this way. I do not think that would cause issues with restarting but I thought I would still share DataBiosphere/terra-docker#476

[LizBaldo]

I will just point my BEE to this branch and see if I can create an AOU runtime and ssh into it

[LizBaldo]

We should probably follow up via slack, but here is what I found out on my BEE:

1- As Qi mentioned above, the issue is not when creating the runtime, but when resuming it
2- When sshing into the VM, I can confirm that the jupyter-server container is running, but the /home/jupyter directory is empty, and there are no jupyter server running, which is why the resume fails. I am not sure what could have caused this quite yet though.

I might need to decouple fixing this from releasing the new bioconductor and gatk images on Terra.

[LizBaldo]

@yonghaoy I think you can close this PR now, #4566 will get you to the latest aou image available

[yonghaoy]

@yonghaoy I think you can close this PR now, #4566 will get you to the latest aou image available

ha! I might just want to delete branch after closing..