Jupyter Enterprise Gateway 3.2.3

Our 3.2.3 release includes the following high-level changes:

• Add support for jinja2 templates when defining KERNEL_POD_NAME
• Expose kernel_info_timeout configuration
• Replace conda with mamba for package management on docker images
• Drop support for Python 3.7
• Multiple dependency security fixes

Thank you to all of our contributors! ❤️ 🎉

What's Changed

• Bump ruff from 0.0.255 to 0.0.257 by @dependabot in #1282
• Bump ruff from 0.0.257 to 0.0.259 by @dependabot in #1283
• Bump ruff from 0.0.259 to 0.0.260 by @dependabot in #1285
• Bump black[jupyter] from 23.1.0 to 23.3.0 by @dependabot in #1284
• [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in #1286
• Bump ruff from 0.0.260 to 0.0.261 by @dependabot in #1288
• Bump ruff from 0.0.261 to 0.0.262 by @dependabot in #1290
• [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in #1293
• Bump ruff from 0.0.262 to 0.0.263 by @dependabot in #1292
• Update docker_swarm.py by @bsdz in #1296
• Bump ruff from 0.0.263 to 0.0.265 by @dependabot in #1298
• Update ruff version in pre-commit config by @kevin-bates in #1299
• Bump ruff from 0.0.265 to 0.0.267 by @dependabot in #1300
• Bump ruff from 0.0.267 to 0.0.269 by @dependabot in #1301
• Bump ruff from 0.0.269 to 0.0.270 by @dependabot in #1303
• [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in #1304
• Update mistune requirement from <1 to <4 by @dependabot in #1306
• Bump actions/checkout from 3 to 4 by @dependabot in #1326
• feat: Add options to mutual authentication by @s4mukka in #1320
• Update link to tensorflow-notebook image by @fresende in #1330
• Bump ruff from 0.0.270 to 0.0.290 by @dependabot in #1331
• Drop support for python 3.7 and upgrade min version of some dependencies by @luong-komorebi in #1334
• [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in #1309
• Bump ruff from 0.0.290 to 0.0.291 by @dependabot in #1335
• docs: reformat list by @IITII in #1350
• [Bug] Fix helm chart logic error by @alex-treebeard in #1352
• Bump github/codeql-action from 2 to 3 by @dependabot in #1354
• Replace conda with mamba on EG docker image by @lresende in #1358
• update crictl version from v1.21.0 to v1.24.0 by @mmmommm in #1364
• Fix readthedocs build configuration by @lresende in #1367
• Bump ruff from 0.0.291 to 0.1.5 by @dependabot in #1348
• Bump nick-invision/retry from 2.8.3 to 3.0.0 by @dependabot in #1365
• Expose kernel_info_timeout configuration by @lresende in #1353
• Add jinja template support to pod-name config via environment variable by @lresende in #1368
• Bump ruff from 0.1.5 to 0.2.2 by @dependabot in #1369
• Bump black[jupyter] from 23.9.1 to 24.2.0 by @dependabot in #1366
• [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in #1340

New Contributors

• @s4mukka made their first contribution in #1320
• @IITII made their first contribution in #1350
• @alex-treebeard made their first contribution in #1352
• @mmmommm made their first contribution in #1364

Full Changelog: v3.2.2...v3.2.3

Jupyter Enterprise Gateway 3.2.2

Our 3.2.2 release includes the following high-level changes:

• Fixes to Spark Operator based kernels to propagate errors on Spark Operator CRD failures
• Update Spark Operator based kernels to set restart strategy to never for spark operator
• Introduces EG_SENSITIVE_ENV_KEYS to filter sensitive values from being logged in ProcessProxy

What's Changed

• Reset SPARK_VERSION to 3.2.1 by @kevin-bates in #1268
• Replace "is defined" in Jinja templates if conditions by @pq43 in #1267
• Update test-install task to build dist as dep by @lresende in #1270
• [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in #1273
• Set restart strategy to never for spark operator by @lresende in #1269
• Bump ruff from 0.0.253 to 0.0.254 by @dependabot in #1272
• Properly propagate errors on Spark Operator CRD failures by @lresende in #1271
• Fix docker swarm not being to start due to missing inherited function on the latest version by @luong-komorebi in #1277
• Updating base image tag to latest available tag by @Chiragasourabh in #1280
• Filter sensitive values from being logged in ProcessProxy by @pq43 in #1279
• Bump ruff from 0.0.254 to 0.0.255 by @dependabot in #1278
• Move shutdown kernel code to be within KernelClient by @pq43 in #1281

New Contributors

• @Chiragasourabh made their first contribution in #1280

Full Changelog: v3.2.1...v3.2.2

Jupyter Enterprise Gateway 3.2.1

Our 3.2.1 release includes the following high-level changes:

• Enhanced Kernel Image Puller with capabilities to pull additional images
• Fixes to Kernel restart corner cases that could cause leaks
• Fixes to Spark Operator based kernels to properly flow environment variables

What's Changed

• Update python interrupt test by @kevin-bates in #1239
• Bump black[jupyter] from 22.10.0 to 22.12.0 by @dependabot in #1236
• Bump ruff from 0.0.189 to 0.0.223 by @dependabot in #1238
• Flow envs to spark operator pods by @kevin-bates in #1242
• Disable healthcheck on kernel-images derived from notebook images by @kevin-bates in #1240
• Add more linting by @blink1073 in #1245
• Update java certs to avoid multi-arch builds to hang by @lresende in #1241
• Dump Kubernetes YAML/CRD on kernel startup fails by @lresende in #1249
• Change demo-base container build from conda to mamba by @lresende in #1250
• Bump black[jupyter] from 22.12.0 to 23.1.0 by @dependabot in #1251
• Bump ruff from 0.0.237 to 0.0.241 by @dependabot in #1252
• [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in #1253
• Bump ruff from 0.0.241 to 0.0.246 by @dependabot in #1254
• A few small changes to gateway client by @pq43 in #1257
• Bump ruff from 0.0.246 to 0.0.249 by @dependabot in #1256
• Fix badges on README by @kevin-bates in #1259
• Fix the flow of kernel-class-name option in bootstrap script by @manoj-jeswani in #1260
• Bump ruff from 0.0.249 to 0.0.253 by @dependabot in #1264
• decouple image name fetching from image puller by @leoyifeng in #1263
• Set language and fix conda env link by @kevin-bates in #1265
• Handle not found exception during remove_kernel by @lresende in #1262

New Contributors

• @pq43 made their first contribution in #1257
• @manoj-jeswani made their first contribution in #1260

Full Changelog: v3.2.0...v3.2.1

Jupyter Enterprise Gateway 3.2.0

Our 3.2.0 release includes the following high-level changes:

• Enhanced support for Spark Operator based kernels
• Helm chart improvements to configuring culling connected kernels

What's Changed

• [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in #1211
• Adopt ruff and address lint by @blink1073 in #1212
• Fix kernel-spark-py and kernel-spark-r images by @kevin-bates in #1217
• delete PATH env conflict with container by @jeremythuon in #1225
• Support building image with dedicated platform by specifying compatible docker flags by @luong-komorebi in #1227
• Bump nick-invision/retry from 2.8.2 to 2.8.3 by @dependabot in #1228
• Start EG in demo-image to inherit PYSPARK_PYTHON env by @kevin-bates in #1230
• Support 'cull connected kernels' as a config in helm and create a way to dynamically pass other environments variables to helm deployment by @luong-komorebi in #1231
• [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in #1229
• Add spell check and ensure docstrings by @blink1073 in #1226
• Fix shutdown of CRD process proxy (and subclasses) by @kevin-bates in #1234

Full Changelog: v3.1.0...v3.2.0

Jupyter Enterprise Gateway 3.1.0

The primary purpose of this release is to cap our dependency on Jupyter Server to < 2.0 since that release requires jupyter_client >= 7 which we can't support.

Since there are some feature-based contributions, we decided to make this a minor release.

Thank you to all of our contributors! ❤️ 🎉

What's Changed

• [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in #1182
• Cap jupyter_server < 2 until kernel provisioners are supported by @kevin-bates in #1186
• Flow kernel-class-name to kernel container if specified in kernel specification by @kevin-bates in #1187
• Add dependabot config by @blink1073 in #1189
• Bump actions/checkout from 2 to 3 by @dependabot in #1190
• Bump nick-invision/retry from 1.0.0 to 2.8.2 by @dependabot in #1191
• multiarch image build support by @leoyifeng in #1194
• [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in #1196
• Allow KIP to pull images relative to kernel provisioners by @kevin-bates in #1197
• Support Kubernetes scheduling constraints ( tolerations, nodeselector and affinity ) and add ingress class name by @luong-komorebi in #1199
• Handle the case when pod and/or pvc has already been created by @luong-komorebi in #1202
• Maintenance cleanup by @blink1073 in #1203
• [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in #1204
• [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in #1205
• Don't perform multi-arch image builds on dry runs by @kevin-bates in #1206
• Update python version, make docker-stacks tags consistent by @kevin-bates in #1207

New Contributors

• @dependabot made their first contribution in #1190
• @leoyifeng made their first contribution in #1194
• @luong-komorebi made their first contribution in #1199

Full Changelog: v3.0.0...v3.1.0

Jupyter Enterprise Gateway 3.0.0

Our 3.0.0 release includes the following high-level changes.

• Support for Spark 3.0 and pod templates
• Improved deployment of new kernels in containerized environments
• Support for CRDs and ConfigMaps on Spark Operator
• Improved helm chart integration
• Extended kernel persistence support
• Role-based documentation: Users, Operators, Contributors, Developers
• Enabled support for images from private registries
• Formalized HA support (singe-instance vs. multi-instance)
• Extend LB algorithms for distributed kernels
• Enabled support for any subclass of IpyKernel kernels
• Enabled support for ConfigMaps in Spark Operator
• Allow operators to increase the maximum sockets and thread pool of the ZMQ context
• Add support for managing service and config map in kernel-pod
• Bug fixes and maintenance

Thank you to all of our contributors!

What's Changed

• Restyle docs to that used in JupyterServer team-compass docs by @kevin-bates in #1034
• Revert the RTD python version to 3.8 by @kevin-bates in #1035
• Fix RTD - remove IPython highlighting reference by @kevin-bates in #1036
• Reorganize documentation by @kevin-bates in #1040
• Drop Python 3.6 and add Python 3.10 to setup metadata and CI matrix by @kevin-bates in #1028
• fix small typo in developer docs by @bloomsa in #1042
• Fix build to propagate SA macro between makefiles by @kevin-bates in #1044
• Fix operator references that were directing to Python docs by @kevin-bates in #1043
• Remove usage of ipython_genutils by @blink1073 in #1046
• Fix links in readme by @icankeep in #1050
• Clean up CI by @blink1073 in #1052
• Docker docs by @bloomsa in #1049
• Fix up some docker image README issues (minor) by @kevin-bates in #1057
• Fix links by @blink1073 in #1056
• Update distributed.py by @LeeMoonCh in #1055
• Adopt pre-commit by @blink1073 in #1061
• Bump jinja2 dependency floor by @kevin-bates in #1064
• Run auto formatters by @blink1073 in #1065
• Add git-blame-ignore-revs by @blink1073 in #1066
• Clean up pytest handling by @blink1073 in #1067
• Add a kernel opening mechanism for load balancing! by @LeeMoonCh in #1068
• Update links to default branch by @blink1073 in #1072
• Add doc entry for load-balancing algorithm configuration by @kevin-bates in #1070
• Support startup scripts and ipykernel subclasses by @kevin-bates in #1076
• Remove conda env config from build workflow by @kevin-bates in #1079
• Ignore deprecation warning from paramiko during pytest execution by @kevin-bates in #1081
• Use flit build backend by @blink1073 in #1083
• fix(k8s): handle status of deleted Pod correctly by @dnwe in #1084
• Attempt to fix FD leak issue #1051 by @rahul26goyal in #1054
• Use hatch and tbump by @blink1073 in #1091
• update helm chart ingress by @darkstarmv in #1073
• Bump paramiko version, remove temp ignore of deprecation warning by @kevin-bates in #1092
• Enable CodeQL analysis in CI by @akchinSTC in #1093
• Add support for Spark 3.0 and pod templates for k8s by @kevin-bates in #559
• Add env var to specify a spark k8s operator configMap by @tahesse in #1087
• Add support to define terminationGracePeriodSeconds in helm chart by @CiprianAnton in #1099
• Release preparation by @kevin-bates in #1102
• Use conda clean --all instead of -tipsy by @kevin-bates in #1103
• Enabling specifying annotations for service accounts in helm chart by @afaytelson in #1106
• reintroduce conda env setup and docs by @bloomsa in #1107
• Refactoring EG Kernel Image Puller (KIP) Python Script by @rahul26goyal in #1090
• Added new WebhookKernelSessionManager for Kernel Persistence by @amazingarman in #1101
• Typo in make file example for running UTs by @rahul26goyal in #1109
• Update dependency requirements to support make env target by @kiersten-stokes in #1108
• Install pre-commit as part of the lint make task by @lresende in #1111
• Add clean-env task to remove conda environment by @lresende in #1110
• Make EnterpriseGatewayApp.shutdown more robust, since it's a cleanup procedure by @CiprianAnton in #1121
• Enable EG to create pods via CRDs on Spark Operator by @lresende in #1119
• Remove references to enterprise-gateway.yaml script by @kevin-bates in #1123
• Fix shutdown issues by @kevin-bates in #1117
• Availability modes by @kevin-bates in #1095
• Update tensorflow-gpu base image by @kevin-bates in #1122
• KIP: Define pulled_images as set, not list by @kevin-bates in #1125
• Switch default to create PSP in KIP to false due to deprecation by @kevin-bates in #1126
• Changes for handling interrupts for PySpark Kubernetes Kernel by @SamRanjha in #1115
• Add type hints to EG application by @kiersten-stokes in #1130
• 851: Add typing hints to process proxies by @bloomsa in #1129
• Enable ability to not filter kernels in container deployments by @kevin-bates in #1131
• Add kernel launch sequence diagram to contributors guide by @kevin-bates in #1132
• Add typing hints to session managers and handler by @kiersten-stokes in #1135
• Enable overriding Helm location from env var by @lresende in #1139
• Rename allowed env configuration traits by @kevin-bates in #1137
• Add type hints to kernels service by @kevin-bates in #1140
• Fix Makefile WHEEL_FILES list by @kevin-bates in #1142
• Remove reference to eg yaml that no longer exists by @kevin-bates in #1144
• Use local kernel images if they exist by @kevin-bates in #1143
• Address delayed shutdown behavior in k8s kernels by @kevin-bates in #1146
• Bump the min version of Python Kubernetes client by @rahul26goyal in #1148
• Ensure kernel session retrieve from resp is correct type by @chiawchen in #1147
• Update v2.6.0 tar file reference by @kevin-bates in #1149
• small API path correction by @bloomsa in #1150
• Add hook to modify zmq.Context by @arv1ndn in #1154
• Update kernel manager library docs by @chiawchen in htt...

Jupyter Enterprise Gateway 3.0.0rc1

Our 3.0.0 release is set to include the following (we'll firm these up as we near GA - hopefully the week of Oct 24):

• Support for Spark 3.0 and pod templates
• Improved deployment of new kernels in containerized environments
• Support for CRDs and ConfigMaps on Spark Operator
• Improved helm chart integration
• Extended kernel persistence support
• Role-based documentation: Users, Operators, Contributors, Developers
• Enabled support for images from private registries
• Formalized HA support (singe-instance vs. multi-instance)
• Extend LB algorithms for distributed kernels
• Enabled support for any subclass of IpyKernel kernels
• Enabled support for ConfigMaps in Spark Operator
• Allow operators to increase the maximum sockets and thread pool of the ZMQ context
• Bug fixes and maintenance

Jupyter Enterprise Gateway 3.0.0rc0

Our 3.0.0 release is set to include the following (we'll firm these up as we near GA):

• Support for Spark 3.0 and pod templates
• Improved deployment of new kernels in containerized environments
• Support for CRDs and ConfigMaps on Spark Operator
• Improved helm chart integration
• Extended kernel persistence support
• Role-based documentation: Users, Operators, Contributors, Developers
• Enabled support for images from private registries
• Formalized HA support (singe-instance vs. multi-instance)
• Extend LB algorithms for distributed kernels
• Enabled support for any subclass of IpyKernel kernels
• Enabled support for ConfigMaps in Spark Operator
• Allow operators to increase the maximum sockets and thread pool of the ZMQ context
• Bug fixes and maintenance

Jupyter Enterprise Gateway 3.0.0b0

Our 3.0.0 release is set to include the following (we'll firm these up as we near GA):

• Support for Spark 3.0 and pod templates
• Improved deployment of new kernels in containerized environments
• Support for CRDs and ConfigMaps on Spark Operator
• Improved helm chart integration
• Extended kernel persistence support
• Role-based documentation: Users, Operators, Contributors, Developers
• Enabled support for images from private registries
• Formalized HA support (singe-instance vs. multi-instance)
• Extend LB algorithms for distributed kernels
• Enabled support for any subclass of IpyKernel kernels
• Enabled support for ConfigMaps in Spark Operator
• Bug fixes and maintenance

Jupyter Enterprise Gateway 2.6.0

Our 2.6.0 release contains the following changes:

Full Change Log

Features

• Use jupyter_server instead of notebook (#927)
• Single Response Address Support (#877)
• Add support for auth token to kernel image puller (#959)
• Allow SSH authentication via GSS. (#950)
• Add crd process proxy to support some crd based k8s applications (#991) ** E X P E R I M E N T A L **
• Normalize kernel-launcher arguments (#988)
• KIP: Add support for containerd-based installations (#1007)
• Add support for YARN API non-anonymous access (#981)
• Enable debugger for python-related kernelspecs, add option to mount kernelspecs with pvc (#986)

Fixes and Maintenance Changes

• Update build status to reflect GH actions (#937)
• Fix port range handling (#952)
• Update broken links to Jupyter pages (#958)
• Fix lazy import of launch_instance (#961)
• Modify the logic that converts EG_LOG_LEVEL to int (#966)
• Add typing hints to kernelspec handler (#965)
• Update docker.md (#973)
• Changes necessary to get itest-docker passing (#964)
• Factor out key_exists helper (#982)
• Update Watchdog dependency to version 2.1.3 (#977)
• Cap jupyter_client < 7.0 (#983)
• Fix make dev to set PYTHONPATH (#993)
• Fix Makefile install target (#994)
• Response port retries (#990)
• Add typing hints to api handler (#1006)
• Add origin restriction config (#997)
• Make KIP sleep when container runtime is not docker (#995)
• Fix kernelspec authorization (#987)
• Fix Spark K8s Kernel Restart issue (#1014)
• Fix issue when log file got rewritten on kernel's restart (#1015)
• Pass self.base_url to Tornado constructor (#1019)
• Update helm deployment instructions in docs (#1026)
• Update base jupyter images, use their jupyter packages (#1029)
• Validate images relative to python, spark, and debugger (#1030)
• Spark operator k8s updates to enable it in gateway on k8s (#1031)
• Build images during release, capture release build/dist dirs (#1032)
• Fix deployment of spark_python_operator (#1033)

Many thanks to the following contributors:

(GitHub contributors page for this release)

@abzymeinsjtu, @akchinSTC, Aleksandr Kandrin, @amarinder-bindra, @bgerrity, @bug-developer021, @camer314, @Carreau, @faurik, @gaocegege, @gogasca, @kevin-bates, @lresende, @rahul26goyal, @ricklamers, @taeyeopkim1, @telamonian, @uniocto, 忆梵