Skip to content

Commit

Permalink
Prepare release 2.5.2
Browse files Browse the repository at this point in the history
  • Loading branch information
kevin-bates committed Dec 9, 2021
1 parent 035cb9e commit 0c97675
Show file tree
Hide file tree
Showing 13 changed files with 41 additions and 41 deletions.
2 changes: 1 addition & 1 deletion Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ SA?=source activate
ENV:=enterprise-gateway-dev
SHELL:=/bin/bash

VERSION?=2.5.2.dev0
VERSION?=2.5.2
SPARK_VERSION?=2.4.6

ifeq (dev, $(findstring dev, $(VERSION)))
Expand Down
4 changes: 2 additions & 2 deletions docs/source/docker.md
Original file line number Diff line number Diff line change
Expand Up @@ -65,7 +65,7 @@ Users that do not wish to extend an existing kernel image must be cognizant of a
2. Is the base image one from [Jupyter Docker-stacks](https://github.com/jupyter/docker-stacks)?

#### Requirements for Custom Kernel Images
Custom kernel images require some support files from the Enterprise Gateway repository. These are packaged into a tar file for each release starting in `2.5.1`. This tar file (named `jupyter_enterprise_gateway_kernel_image_files-VERSION.tar.gz`) is composed of a few files - one bootstrap script and a kernel launcher (one per kernel type).
Custom kernel images require some support files from the Enterprise Gateway repository. These are packaged into a tar file for each release starting in `2.5.2`. This tar file (named `jupyter_enterprise_gateway_kernel_image_files-VERSION.tar.gz`) is composed of a few files - one bootstrap script and a kernel launcher (one per kernel type).

##### Bootstrap-kernel.sh
Enterprise Gateway provides a single [bootstrap-kernel.sh](https://github.com/jupyter/enterprise_gateway/blob/master/etc/kernel-launchers/bootstrap/bootstrap-kernel.sh) script that handles the three kernel languages supported out of the box - Python, R, and Scala. When a kernel image is started by Enterprise Gateway, parameters used within the bootstrap-kernel.sh script are conveyed via environment variables. The bootstrap script is then responsible for validating and converting those parameters to meaningful arguments to the appropriate launcher.
Expand Down Expand Up @@ -119,7 +119,7 @@ If your base image is not from docker-stacks, it is recommended that you NOT run
Aside from configuring the image user, all other aspects of customization are the same. In this case, we'll use the tensorflow-gpu image and convert it to be usable via Enterprise Gateway as a custom kernel image. Note that because this image didn't have `wget` we used `curl` to download the supporting kernel-image files.

```dockerfile
FROM tensorflow/tensorflow:2.5.1-gpu-jupyter
FROM tensorflow/tensorflow:2.5.2-gpu-jupyter

USER root

Expand Down
2 changes: 1 addition & 1 deletion docs/source/getting-started.md
Original file line number Diff line number Diff line change
Expand Up @@ -148,7 +148,7 @@ Note that because the Apache Toree kernel, and its supporting libraries, will be

#### Sample kernelspecs

We provide sample kernel configuration and launcher tar files as part of [each release](https://github.com/jupyter/enterprise_gateway/releases) (e.g. [jupyter_enterprise_gateway_kernelspecs-2.5.1.tar.gz](https://github.com/jupyter/enterprise_gateway/releases/download/v2.5.1/jupyter_enterprise_gateway_kernelspecs-2.5.1.tar.gz)) that can be extracted and modified to fit your configuration.
We provide sample kernel configuration and launcher tar files as part of [each release](https://github.com/jupyter/enterprise_gateway/releases) (e.g. [jupyter_enterprise_gateway_kernelspecs-2.5.2.tar.gz](https://github.com/jupyter/enterprise_gateway/releases/download/v2.5.2/jupyter_enterprise_gateway_kernelspecs-2.5.2.tar.gz)) that can be extracted and modified to fit your configuration.

For information about how to build your own kernel-based docker image for use by Enterprise Gateway see [Custom kernel images](docker.html#custom-kernel-images).

Expand Down
4 changes: 2 additions & 2 deletions docs/source/kernel-distributed.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,9 +9,9 @@ The following sample kernelspecs are currently available on Distributed mode:
Install the `python_distributed` kernelspec on all nodes.

``` Bash
wget https://github.com/jupyter/enterprise_gateway/releases/download/v2.5.1/jupyter_enterprise_gateway_kernelspecs-2.5.1.tar.gz
wget https://github.com/jupyter/enterprise_gateway/releases/download/v2.5.2/jupyter_enterprise_gateway_kernelspecs-2.5.2.tar.gz
KERNELS_FOLDER=/usr/local/share/jupyter/kernels
tar -zxvf jupyter_enterprise_gateway_kernelspecs-2.5.1.tar.gz --strip 1 --directory $KERNELS_FOLDER/python_distributed/ python_distributed/
tar -zxvf jupyter_enterprise_gateway_kernelspecs-2.5.2.tar.gz --strip 1 --directory $KERNELS_FOLDER/python_distributed/ python_distributed/
```

The `python_distributed` kernelspec uses `DistributedProcessProxy` which is responsible for the launch and management of kernels distributed across and explicitly defined set of hosts using ssh. Hosts are determined via a round-robin algorithm (that we should make pluggable someday).
Expand Down
6 changes: 3 additions & 3 deletions docs/source/kernel-kubernetes.md
Original file line number Diff line number Diff line change
Expand Up @@ -523,7 +523,7 @@ By default, _vanilla_ kernels use a value of `none` for the spark context initia

When the kernel is intended to target _Spark-on-kubernetes_, its launch is very much like kernels launched in YARN _cluster mode_, albeit with a completely different set of parameters. Here's an example `SPARK_OPTS` string value which best conveys the idea:
```
"SPARK_OPTS": "--master k8s://https://${KUBERNETES_SERVICE_HOST}:${KUBERNETES_SERVICE_PORT} --deploy-mode cluster --name ${KERNEL_USERNAME}-${KERNEL_ID} --conf spark.kubernetes.driver.label.app=enterprise-gateway --conf spark.kubernetes.driver.label.kernel_id=${KERNEL_ID} --conf spark.kubernetes.executor.label.app=enterprise-gateway --conf spark.kubernetes.executor.label.kernel_id=${KERNEL_ID} --conf spark.kubernetes.driver.docker.image=${KERNEL_IMAGE} --conf spark.kubernetes.executor.docker.image=kubespark/spark-executor-py:v2.5.1-kubernetes-0.5.0 --conf spark.kubernetes.submission.waitAppCompletion=false",
"SPARK_OPTS": "--master k8s://https://${KUBERNETES_SERVICE_HOST}:${KUBERNETES_SERVICE_PORT} --deploy-mode cluster --name ${KERNEL_USERNAME}-${KERNEL_ID} --conf spark.kubernetes.driver.label.app=enterprise-gateway --conf spark.kubernetes.driver.label.kernel_id=${KERNEL_ID} --conf spark.kubernetes.executor.label.app=enterprise-gateway --conf spark.kubernetes.executor.label.kernel_id=${KERNEL_ID} --conf spark.kubernetes.driver.docker.image=${KERNEL_IMAGE} --conf spark.kubernetes.executor.docker.image=kubespark/spark-executor-py:v2.5.2-kubernetes-0.5.0 --conf spark.kubernetes.submission.waitAppCompletion=false",
```
Note that each of the labels previously discussed are also applied to the _driver_ and _executor_ pods.

Expand Down Expand Up @@ -595,9 +595,9 @@ From anywhere with Helm cluster access, create the service and deployment by run
```bash
helm upgrade --install --atomic --namespace enterprise-gateway enterprise-gateway etc/kubernetes/helm/enterprise-gateway
```
the helm chart tarball is also accessible as an asset on our [release](https://github.com/jupyter/enterprise_gateway/releases/tag/v2.5.1) page:
the helm chart tarball is also accessible as an asset on our [release](https://github.com/jupyter/enterprise_gateway/releases/tag/v2.5.2) page:
```bash
helm install --name enterprise-gateway --atomic --namespace enterprise-gateway https://github.com/jupyter/enterprise_gateway/releases/download/v2.5.1/jupyter_enterprise_gateway_helm-2.5.1.tgz
helm install --name enterprise-gateway --atomic --namespace enterprise-gateway https://github.com/jupyter/enterprise_gateway/releases/download/v2.5.2/jupyter_enterprise_gateway_helm-2.5.2.tgz
```
##### Configuration

Expand Down
16 changes: 8 additions & 8 deletions docs/source/kernel-spark-standalone.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,14 +23,14 @@ EG_REMOTE_HOSTS=elyra-node-1.fyre.ibm.com,elyra-node-2.fyre.ibm.com,elyra-node-3
Although Enterprise Gateway does not currently provide sample kernelspecs for Spark standalone, here are the steps necessary to convert a yarn_client kernelspec to standalone.

For each supported Jupyter Kernel, we have provided sample kernel configurations and launchers as part of the release
[jupyter_enterprise_gateway_kernelspecs-2.5.1.tar.gz](https://github.com/jupyter/enterprise_gateway/releases/download/v2.5.1/jupyter_enterprise_gateway_kernelspecs-2.5.1.tar.gz).
[jupyter_enterprise_gateway_kernelspecs-2.5.2.tar.gz](https://github.com/jupyter/enterprise_gateway/releases/download/v2.5.2/jupyter_enterprise_gateway_kernelspecs-2.5.2.tar.gz).

Considering we would like to enable the IPython Kernel that comes pre-installed with Anaconda to run on
Spark Standalone, we would have to copy the sample configuration folder **spark_python_yarn_client**
to where the Jupyter kernels are installed (e.g. jupyter kernelspec list) and rename it to **spark_python_spark_standalone***

``` Bash
wget https://github.com/jupyter/enterprise_gateway/releases/download/v2.5.1/jupyter_enterprise_gateway_kernelspecs-2.5.1.tar.gz
wget https://github.com/jupyter/enterprise_gateway/releases/download/v2.5.2/jupyter_enterprise_gateway_kernelspecs-2.5.2.tar.gz
SCALA_KERNEL_DIR="$(jupyter kernelspec list | grep -w "python3" | awk '{print $2}')"
KERNELS_FOLDER="$(dirname "${SCALA_KERNEL_DIR}")"
tar -zxvf enterprise_gateway_kernelspecs.tar.gz --strip 1 --directory $KERNELS_FOLDER/spark_python_yarn_client/ spark_python_yarn_client/
Expand Down Expand Up @@ -85,9 +85,9 @@ Follow the steps below to install/configure the Toree kernel:
Considering we would like to enable the Scala Kernel to run on YARN Cluster and Client mode we would have to copy the sample configuration folder **spark_scala_yarn_client** to where the Jupyter kernels are installed (e.g. jupyter kernelspec list)

``` Bash
wget https://github.com/jupyter/enterprise_gateway/releases/download/v2.5.1/jupyter_enterprise_gateway_kernelspecs-2.5.1.tar.gz
wget https://github.com/jupyter/enterprise_gateway/releases/download/v2.5.2/jupyter_enterprise_gateway_kernelspecs-2.5.2.tar.gz
KERNELS_FOLDER=/usr/local/share/jupyter/kernels
tar -zxvf jupyter_enterprise_gateway_kernelspecs-2.5.1.tar.gz --strip 1 --directory $KERNELS_FOLDER/spark_scala_yarn_client/ spark_scala_yarn_client/
tar -zxvf jupyter_enterprise_gateway_kernelspecs-2.5.2.tar.gz --strip 1 --directory $KERNELS_FOLDER/spark_scala_yarn_client/ spark_scala_yarn_client/
mv $KERNELS_FOLDER/spark_scala_yarn_client $KERNELS_FOLDER/spark_scala_spark_standalone
```

Expand All @@ -102,9 +102,9 @@ The IPython kernel comes pre-installed with Anaconda and we have tested with its
Considering we would like to enable the IPython kernel to run on YARN Cluster and Client mode we would have to copy the sample configuration folder **spark_python_yarn_client** to where the Jupyter kernels are installed (e.g. jupyter kernelspec list)

``` Bash
wget https://github.com/jupyter/enterprise_gateway/releases/download/v2.5.1/jupyter_enterprise_gateway_kernelspecs-2.5.1.tar.gz
wget https://github.com/jupyter/enterprise_gateway/releases/download/v2.5.2/jupyter_enterprise_gateway_kernelspecs-2.5.2.tar.gz
KERNELS_FOLDER=/usr/local/share/jupyter/kernels
tar -zxvf jupyter_enterprise_gateway_kernelspecs-2.5.1.tar.gz --strip 1 --directory $KERNELS_FOLDER/spark_python_yarn_client/ spark_python_yarn_client/
tar -zxvf jupyter_enterprise_gateway_kernelspecs-2.5.2.tar.gz --strip 1 --directory $KERNELS_FOLDER/spark_python_yarn_client/ spark_python_yarn_client/
mv $KERNELS_FOLDER/spark_python_yarn_client $KERNELS_FOLDER/spark_python_spark_standalone
```

Expand Down Expand Up @@ -137,9 +137,9 @@ ls $ANACONDA_HOME/lib/R/library
Considering we would like to enable the IRkernel to run on YARN Cluster and Client mode we would have to copy the sample configuration folder **spark_R_yarn_client** to where the Jupyter kernels are installed (e.g. jupyter kernelspec list)

``` Bash
wget https://github.com/jupyter/enterprise_gateway/releases/download/v2.5.1/jupyter_enterprise_gateway_kernelspecs-2.5.1.tar.gz
wget https://github.com/jupyter/enterprise_gateway/releases/download/v2.5.2/jupyter_enterprise_gateway_kernelspecs-2.5.2.tar.gz
KERNELS_FOLDER=/usr/local/share/jupyter/kernels
tar -zxvf jupyter_enterprise_gateway_kernelspecs-2.5.1.tar.gz --strip 1 --directory $KERNELS_FOLDER/spark_R_yarn_client/ spark_R_yarn_client/
tar -zxvf jupyter_enterprise_gateway_kernelspecs-2.5.2.tar.gz --strip 1 --directory $KERNELS_FOLDER/spark_R_yarn_client/ spark_R_yarn_client/
mv $KERNELS_FOLDER/spark_R_yarn_client $KERNELS_FOLDER/spark_R_spark_standalone
```

Expand Down
16 changes: 8 additions & 8 deletions docs/source/kernel-yarn-client-mode.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,14 +27,14 @@ EG_REMOTE_HOSTS=elyra-node-1.fyre.ibm.com,elyra-node-2.fyre.ibm.com,elyra-node-3
```

For each supported Jupyter Kernel, we have provided sample kernel configurations and launchers as part of the release
[jupyter_enterprise_gateway_kernelspecs-2.5.1.tar.gz](https://github.com/jupyter/enterprise_gateway/releases/download/v2.5.1/jupyter_enterprise_gateway_kernelspecs-2.5.1.tar.gz).
[jupyter_enterprise_gateway_kernelspecs-2.5.2.tar.gz](https://github.com/jupyter/enterprise_gateway/releases/download/v2.5.2/jupyter_enterprise_gateway_kernelspecs-2.5.2.tar.gz).

Considering we would like to enable the IPython Kernel that comes pre-installed with Anaconda to run on
Yarn Client mode, we would have to copy the sample configuration folder **spark_python_yarn_client**
to where the Jupyter kernels are installed (e.g. jupyter kernelspec list)

``` Bash
wget https://github.com/jupyter/enterprise_gateway/releases/download/v2.5.1/jupyter_enterprise_gateway_kernelspecs-2.5.1.tar.gz
wget https://github.com/jupyter/enterprise_gateway/releases/download/v2.5.2/jupyter_enterprise_gateway_kernelspecs-2.5.2.tar.gz
SCALA_KERNEL_DIR="$(jupyter kernelspec list | grep -w "python3" | awk '{print $2}')"
KERNELS_FOLDER="$(dirname "${SCALA_KERNEL_DIR}")"
tar -zxvf enterprise_gateway_kernelspecs.tar.gz --strip 1 --directory $KERNELS_FOLDER/spark_python_yarn_client/ spark_python_yarn_client/
Expand Down Expand Up @@ -82,9 +82,9 @@ Follow the steps below to install/configure the Toree kernel:
Considering we would like to enable the Scala Kernel to run on YARN Cluster and Client mode we would have to copy the sample configuration folder **spark_scala_yarn_client** to where the Jupyter kernels are installed (e.g. jupyter kernelspec list)

``` Bash
wget https://github.com/jupyter/enterprise_gateway/releases/download/v2.5.1/jupyter_enterprise_gateway_kernelspecs-2.5.1.tar.gz
wget https://github.com/jupyter/enterprise_gateway/releases/download/v2.5.2/jupyter_enterprise_gateway_kernelspecs-2.5.2.tar.gz
KERNELS_FOLDER=/usr/local/share/jupyter/kernels
tar -zxvf jupyter_enterprise_gateway_kernelspecs-2.5.1.tar.gz --strip 1 --directory $KERNELS_FOLDER/spark_scala_yarn_client/ spark_scala_yarn_client/
tar -zxvf jupyter_enterprise_gateway_kernelspecs-2.5.2.tar.gz --strip 1 --directory $KERNELS_FOLDER/spark_scala_yarn_client/ spark_scala_yarn_client/
```

For more information about the Scala kernel, please visit the [Apache Toree](https://toree.apache.org/) page.
Expand All @@ -98,9 +98,9 @@ The IPython kernel comes pre-installed with Anaconda and we have tested with its
Considering we would like to enable the IPython kernel to run on YARN Cluster and Client mode we would have to copy the sample configuration folder **spark_python_yarn_client** to where the Jupyter kernels are installed (e.g. jupyter kernelspec list)

``` Bash
wget https://github.com/jupyter/enterprise_gateway/releases/download/v2.5.1/jupyter_enterprise_gateway_kernelspecs-2.5.1.tar.gz
wget https://github.com/jupyter/enterprise_gateway/releases/download/v2.5.2/jupyter_enterprise_gateway_kernelspecs-2.5.2.tar.gz
KERNELS_FOLDER=/usr/local/share/jupyter/kernels
tar -zxvf jupyter_enterprise_gateway_kernelspecs-2.5.1.tar.gz --strip 1 --directory $KERNELS_FOLDER/spark_python_yarn_client/ spark_python_yarn_client/
tar -zxvf jupyter_enterprise_gateway_kernelspecs-2.5.2.tar.gz --strip 1 --directory $KERNELS_FOLDER/spark_python_yarn_client/ spark_python_yarn_client/
```

For more information about the IPython kernel, please visit the [IPython kernel](https://ipython.readthedocs.io/en/stable/) page.
Expand Down Expand Up @@ -132,9 +132,9 @@ ls $ANACONDA_HOME/lib/R/library
Considering we would like to enable the IRkernel to run on YARN Cluster and Client mode we would have to copy the sample configuration folder **spark_R_yarn_client** to where the Jupyter kernels are installed (e.g. jupyter kernelspec list)

``` Bash
wget https://github.com/jupyter/enterprise_gateway/releases/download/v2.5.1/jupyter_enterprise_gateway_kernelspecs-2.5.1.tar.gz
wget https://github.com/jupyter/enterprise_gateway/releases/download/v2.5.2/jupyter_enterprise_gateway_kernelspecs-2.5.2.tar.gz
KERNELS_FOLDER=/usr/local/share/jupyter/kernels
tar -zxvf jupyter_enterprise_gateway_kernelspecs-2.5.1.tar.gz --strip 1 --directory $KERNELS_FOLDER/spark_R_yarn_client/ spark_R_yarn_client/
tar -zxvf jupyter_enterprise_gateway_kernelspecs-2.5.2.tar.gz --strip 1 --directory $KERNELS_FOLDER/spark_R_yarn_client/ spark_R_yarn_client/
```

For more information about the iR kernel, please visit the [IRkernel](https://irkernel.github.io/) page.
Expand Down
Loading

0 comments on commit 0c97675

Please sign in to comment.