Ia 4362 python upgrade #434

lucymcnatt · 2023-06-23T20:39:16Z

Jira ticket: https://broadworkbench.atlassian.net/browse/IA-4364
Jira ticket: https://broadworkbench.atlassian.net/browse/IA-3994
Jira ticket: https://broadworkbench.atlassian.net/browse/IA-4271
Jira ticket: https://broadworkbench.atlassian.net/browse/IA-4304

Summary of changes

What

Updates the base image from tf-gpu.2-7 to tf-gpu.2-11.py310 to use python 3.10
Removes references to gatk-otvf image
Enables toc2 jupyter extension to allow file downloads in different formats

Why

Python 3.7 is EOL (no more bug patches) at the end of June
Intel decided to stop development of OVTF and will no longer be maintained
Users want to save their notebooks as HTML or other formats

.github/workflows/test-terra-jupyter-python.yml

config/conf.json

lucymcnatt · 2023-06-29T13:37:36Z

terra-jupyter-aou/Dockerfile

@@ -48,7 +45,7 @@ ENV PYTHONPATH $PYTHONPATH:/usr/lib/spark/python

 ENV PYSPARK_PYTHON=python3

-ENV HAIL_VERSION=0.2.107
+ENV HAIL_VERSION=0.2.109


keeping this up to date with the hail dockerfile

lucymcnatt · 2023-06-29T13:38:02Z

terra-jupyter-aou/Dockerfile

@@ -264,8 +261,6 @@ RUN curl -L http://www.repeatmasker.org/RepeatMasker/RepeatMasker-4.1.4.tar.gz \
 RUN rm -f /usr/local/bin/workspace_cromwell.py
 # alias cromshell_beta to cromshell
 RUN echo "alias cromshell='cromshell-beta'" >> ~/.bash_aliases
-# Downgrade jupyter_contrib_nbextensions as the newer version breaks snippet_menu extension
-RUN pip install --force-reinstall -v "jupyter_contrib_nbextensions==0.5.1"


downgrading this broke nbconvert

You are saying that there is no need to downgrade anymore?

as far as I know? It built fine, but I was going to maybe ask Yonghao about how to test that this extension is working when testing

terra-jupyter-base/Dockerfile

lucymcnatt · 2023-06-29T13:41:04Z

terra-jupyter-base/Dockerfile

-# verify cromshell 2.0 is installed
-RUN cromshell-beta version
+# to allow file-saving in different formats
+RUN jupyter nbextension enable toc2/main


for file format ticket

lucymcnatt · 2023-06-29T13:43:51Z

terra-jupyter-r/Dockerfile

 USER root

 COPY scripts $JUPYTER_HOME/scripts

 # Add env vars to identify binary package installation
-ENV TERRA_R_PLATFORM="terra-jupyter-r"
+ENV TERRA_R_PLATFORM="terra-jupyter-r-1.1.0"


changed env var to include version to resolve issue with binary package installation, thread here

terra-jupyter-python/tests/smoke_test.ipynb

LizBaldo

LGTM! Thanks a bunch for the huge amount of work and testing! I mostly have a few request to document package changes in the code, and trying to rerun the flaky failed test

LizBaldo · 2023-06-29T18:15:48Z

README.md


-It is not advised to run build.sh locally, as this will push to the remote docker repo and delete the image locally upon completion.  
+If you're on an M1 and building an image from a locally built image, replace the current FROM command:


LizBaldo · 2023-06-29T18:16:13Z

README.md

@@ -103,7 +109,7 @@ Running locally is conventient for quick development and exploring the image. Ho
 - there are no environment variables like `GOOGLE_PROJECT`, `WORKSPACE_NAME`, `WORKSPACE_BUCKET`, etc when running locally
 - there is no workspace-syncing when run locally

-To launch an image through Terra, navigate to https://app.terra.bio, select a workspace, enter your image in the "Custom Image" field, and click Create.
+To launch an image through Terra, navigate to https://app.terra.bio or your BEE's UI, select a workspace, enter your new image in the "Custom Image" field, and click Create.


So happy this works!

config/conf.json

LizBaldo · 2023-06-29T18:20:35Z

config/legacy_static_images.json

@@ -1,11 +1,20 @@
 [
  {
-    "updated": "2023-02-09",
-    "image": "us.gcr.io/broad-dsp-gcr-public/terra-jupyter-bioconductor:2.1.9",
+    "updated": "2023-06-23",


Ohh I think I might have forgotten to update these last time, but it is a legacy_static file so do we need to do this? Regardless it should be 2.1.11 for the bioconductor image I think

oh yes definitely 2.1.11! and updating these to the most recent version before the python 3.10 release was something Alessandro suggested so that users can select those if they still want 3.7. Also in the CONTRIBUTING.md it says:

The version of these legacy images is the version prior to a major or minor version bump

LizBaldo · 2023-06-29T18:22:29Z

terra-jupyter-aou/Dockerfile

@@ -264,8 +261,6 @@ RUN curl -L http://www.repeatmasker.org/RepeatMasker/RepeatMasker-4.1.4.tar.gz \
 RUN rm -f /usr/local/bin/workspace_cromwell.py
 # alias cromshell_beta to cromshell
 RUN echo "alias cromshell='cromshell-beta'" >> ~/.bash_aliases
-# Downgrade jupyter_contrib_nbextensions as the newer version breaks snippet_menu extension
-RUN pip install --force-reinstall -v "jupyter_contrib_nbextensions==0.5.1"


You are saying that there is no need to downgrade anymore?

terra-jupyter-base/Dockerfile

config/legacy_static_images.json

lucymcnatt added 23 commits June 22, 2023 13:47

[IA-4362] Upgrade to python 3.10 and remove gatk-ovtf

fd14a41

updating versions

66bd55c

removing minor version

8d5ba76

strings around 3.10

16a4d2a

more strings

04a3eb5

remove rm nvidia

f4f0f8e

py310 tf-gpu

873de10

cromshell fix

5582514

add back in packages

38a2504

updating protobuf to align with deep learning image

26ca73c

rich already installed, changing test package

a93c4e6

docstring-parser already installed, changing test package

e5d8117

actually install the package;

d9908fd

move print statement

3162242

updating requirements

494cfae

removing google-auth, adding bq to solve dependency probs

6d9fb2b

move assert back into print block

f265dd0

remove keras install

88c7469

disable binaries

deecf6a

change terra-platform env var

4920a9e

oops, local version

15fb4bd

enable toc2 and remove python3.10-dev

6eabb8a

dont downgrade contrib_nbextensions

9bef113

lucymcnatt marked this pull request as ready for review June 29, 2023 13:33

lucymcnatt requested review from LizBaldo and jdcanas June 29, 2023 13:33