Maintenance cleanup (#1171)

* maintenance cleanup
* add rollup check
* Minor formatting fixups

Co-authored-by: Kevin Bates <kbates4@gmail.com>

.github/ISSUE_TEMPLATE.md

@@ -10,6 +10,6 @@ To generate better logs, please run the gateway with `--debug` command line para
 
 ## Environment
 
-- Enterprise Gateway Version [e.g. 1.x, 2.x, ...]
-- Platform: [e.g. YARN, Kubernetes ...]
-- Others [e.g. Jupyter Server 5.7, JupyterHub 1.0, etc]
+- Enterprise Gateway Version \[e.g. 1.x, 2.x, ...\]
+- Platform: \[e.g. YARN, Kubernetes ...\]
+- Others \[e.g. Jupyter Server 5.7, JupyterHub 1.0, etc\]

.github/workflows/build.yml

@@ -13,7 +13,7 @@ jobs:
       fail-fast: false
       matrix:
         os: [ubuntu-latest]
-        python-version: ["3.7", "3.8", "3.9", "3.10", "3.11.0-beta - 3.11.0"]
+        python-version: ["3.7", "3.8", "3.9", "3.10", "3.11-dev"]
     steps:
       - name: Checkout
         uses: actions/checkout@v2
@@ -61,8 +61,14 @@ jobs:
           docker logs itest-yarn
       - name: Bump versions
         run: |
-          pip install tbump
-          tbump --dry-run --no-tag --no-push 100.100.100rc0
+          pipx run tbump --dry-run --no-tag --no-push 100.100.100rc0
+
+  pre_commit:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v2
+      - uses: jupyterlab/maintainer-tools/.github/actions/base-setup@v1
+      - uses: jupyterlab/maintainer-tools/.github/actions/pre-commit@v1
 
   link_check:
     runs-on: ubuntu-latest
@@ -116,3 +122,20 @@ jobs:
     steps:
       - uses: jupyterlab/maintainer-tools/.github/actions/base-setup@v1
       - uses: jupyterlab/maintainer-tools/.github/actions/test-sdist@v1
+
+
+  python_tests_check: # This job does nothing and is only used for the branch protection
+    if: always()
+    needs:
+      - build
+      - link_check
+      - test_minimum_versions
+      - build_docs
+      - pre_commit
+      - test_sdist
+    runs-on: ubuntu-latest
+    steps:
+      - name: Decide whether the needed jobs succeeded or failed
+        uses: re-actors/alls-green@release/v1
+        with:
+          jobs: ${{ toJSON(needs) }}

.pre-commit-config.yaml

@@ -31,15 +31,16 @@ repos:
         files: \.py$
         args: [--profile=black]
 
-  - repo: https://github.com/pre-commit/mirrors-prettier
-    rev: v3.0.0-alpha.0
+  - repo: https://github.com/abravalheri/validate-pyproject
+    rev: v0.10.1
     hooks:
-      - id: prettier
-        exclude: |
-          (?x)^(
-            etc/kubernetes/.*.yaml|
-            website/.*
-          )$
+      - id: validate-pyproject
+        stages: [manual]
+
+  - repo: https://github.com/executablebooks/mdformat
+    rev: 0.7.16
+    hooks:
+      - id: mdformat
 
   - repo: https://github.com/asottile/pyupgrade
     rev: v2.38.2
@@ -53,12 +54,14 @@ repos:
       - id: doc8
         args: [--max-line-length=200]
 
-  - repo: https://github.com/pycqa/flake8
-    rev: 5.0.4
+  - repo: https://github.com/john-hen/Flake8-pyproject
+    rev: 1.0.1
     hooks:
-      - id: flake8
+      - id: Flake8-pyproject
+        alias: flake8
         additional_dependencies:
           ["flake8-bugbear==22.6.22", "flake8-implicit-str-concat==0.2.0"]
+        stages: [manual]
 
   - repo: https://github.com/sirosen/check-jsonschema
     rev: 0.18.3

LICENSE.md

@@ -57,5 +57,7 @@ change to one of the Jupyter repositories.
 With this in mind, the following banner should be used in any source code file
 to indicate the copyright and license terms:
 
-    # Copyright (c) Jupyter Development Team.
-    # Distributed under the terms of the Modified BSD License.
+```
+# Copyright (c) Jupyter Development Team.
+# Distributed under the terms of the Modified BSD License.
+```

README.md

@@ -33,8 +33,8 @@ an enterprise. Inspired by Jupyter Kernel Gateway, Jupyter Enterprise Gateway pr
 
 - Adds support for remote kernels hosted throughout the enterprise where kernels can be launched in
   the following ways:
-  _ Local to the Enterprise Gateway server (today's Kernel Gateway behavior)
-  _ On specific nodes of the cluster utilizing a round-robin algorithm \* On nodes identified by an associated resource manager
+  \_ Local to the Enterprise Gateway server (today's Kernel Gateway behavior)
+  \_ On specific nodes of the cluster utilizing a round-robin algorithm * On nodes identified by an associated resource manager
 - Provides support for Apache Spark managed by YARN, IBM Spectrum Conductor, Kubernetes or Docker Swarm out of the box. Others can be configured via Enterprise Gateway's extensible framework.
 - Secure communication from the client, through the Enterprise Gateway server, to the kernels
 - Multi-tenant capabilities

docs/source/contributors/system-architecture.md

@@ -32,9 +32,9 @@ environment variables (also described in the kernel specification), is passed to
 a process class. This class supports four basic methods following its creation:
 
 1. `poll()` to determine if the process is still running
-2. `wait()` to block the caller until the process has terminated
-3. `send_signal(signum)` to send a signal to the process
-4. `kill()` to terminate the process
+1. `wait()` to block the caller until the process has terminated
+1. `send_signal(signum)` to send a signal to the process
+1. `kill()` to terminate the process
 
 As you can see, other forms of process communication can be achieved by abstracting the launch mechanism.
 
@@ -83,7 +83,7 @@ See the [Process Proxy](#process-proxy) section for more details on process prox
 `RemoteMappingKernelManager` is a subclass of Jupyter Server's [`AsyncMappingKernelManager`](https://github.com/jupyter-server/jupyter_server/blob/745f5ba3f00280c1e1900326a7e08463d48a3912/jupyter_server/services/kernels/kernelmanager.py#L633) and provides two functions.
 
 1. It provides the vehicle for making the `RemoteKernelManager` class known and available.
-2. It overrides `start_kernel` to look at the target kernel's kernel spec to see if it contains a remote process proxy class entry. If so, it records the name of the class in its member variable to be made available to the kernel start logic.
+1. It overrides `start_kernel` to look at the target kernel's kernel spec to see if it contains a remote process proxy class entry. If so, it records the name of the class in its member variable to be made available to the kernel start logic.
 
 ## Remote Kernel Manager
 
@@ -404,9 +404,9 @@ Kernel launchers provide a means of normalizing behaviors across kernels while a
 There are four primary tasks of a kernel launcher:
 
 1. Creation of the connection file and ZMQ ports on the remote (target) system along with a _gateway listener_ socket
-2. Conveyance of the connection (and listener socket) information back to the Enterprise Gateway process
-3. Invocation of the target kernel
-4. Listen for interrupt and shutdown requests from Enterprise Gateway and carry out the action when appropriate
+1. Conveyance of the connection (and listener socket) information back to the Enterprise Gateway process
+1. Invocation of the target kernel
+1. Listen for interrupt and shutdown requests from Enterprise Gateway and carry out the action when appropriate
 
 Kernel launchers are minimally invoked with three parameters (all of which are conveyed by the `argv` stanza of the corresponding `kernel.json` file) - the kernel's ID as created by the server and conveyed via the placeholder `{kernel_id}`, a response address consisting of the Enterprise Gateway server IP and port on which to return the connection information similarly represented by the placeholder `{response_address}`, and a public-key used by the launcher to encrypt an AES key that encrypts the kernel's connection information back to the server and represented by the placeholder `{public_key}`.
 
@@ -447,6 +447,7 @@ Here's a [kernel.json](https://github.com/jupyter-server/enterprise_gateway/blob
 Other options supported by launchers include:
 
 - `--RemoteProcessProxy.port-range {port_range}` - passes configured port-range to launcher where launcher applies that range to kernel ports. The port-range may be configured globally or on a per-kernel specification basis, as previously described.
+
 - `--RemoteProcessProxy.spark-context-initialization-mode [lazy|eager|none]` - indicates the _timeframe_ in which the spark context will be created.
 
   - `lazy` (default) attempts to defer initialization as late as possible - although this can vary depending on the
@@ -479,11 +480,11 @@ eval exec \
 Theoretically speaking, enabling a kernel for use in other frameworks amounts to the following:
 
 1. Build a kernel specification file that identifies the process proxy class to be used.
-2. Implement the process proxy class such that it supports the four primitive functions of
+1. Implement the process proxy class such that it supports the four primitive functions of
    `poll()`, `wait()`, `send_signal(signum)` and `kill()` along with `launch_process()`.
-3. If the process proxy corresponds to a remote process, derive the process proxy class from
+1. If the process proxy corresponds to a remote process, derive the process proxy class from
    `RemoteProcessProxy` and implement `confirm_remote_startup()` and `handle_timeout()`.
-4. Insert invocation of a launcher (if necessary) which builds the connection file and
+1. Insert invocation of a launcher (if necessary) which builds the connection file and
    returns its contents on the `{response_address}` socket and following the encryption protocol set forth in the other launchers.
 
 ```{seealso}

docs/source/developers/custom-images.md

@@ -21,7 +21,7 @@ USER $NB_UID  # switch back to the jovyan user
 Users that do not wish to extend an existing kernel image must be cognizant of a couple of things.
 
 1. Requirements of a kernel-based image to be used by Enterprise Gateway.
-2. Is the base image one from [Jupyter Docker-stacks](https://github.com/jupyter/docker-stacks)?
+1. Is the base image one from [Jupyter Docker-stacks](https://github.com/jupyter/docker-stacks)?
 
 ### Requirements for Custom Kernel Images
 

docs/source/developers/dev-process-proxy.md

@@ -19,10 +19,10 @@ That said, if you and your organization plan to stay on Enterprise Gateway 2.x o
 Please refer to the [Process Proxy section](../contributors/system-architecture.md#process-proxy) in the System Architecture pages for descriptions and structure of existing process proxies. Here is the general guideline for the process of implementing a process proxy.
 
 1. Identify and understand how to _decorate_ your "job" within the resource manager. In Hadoop YARN, this is done by using the kernel's ID as the _application name_ by setting the [`--name` parameter to `${KERNEL_ID}`](https://github.com/jupyter-server/enterprise_gateway/blob/54c8e31d9b17418f35454b49db691d2ce5643c22/etc/kernelspecs/spark_python_yarn_cluster/kernel.json#L14). In Kubernetes, we apply the kernel's ID to the [`kernel-id` label on the POD](https://github.com/jupyter-server/enterprise_gateway/blob/54c8e31d9b17418f35454b49db691d2ce5643c22/etc/kernel-launchers/kubernetes/scripts/kernel-pod.yaml.j2#L16).
-2. Today, all invocations of kernels into resource managers use a shell or python script mechanism configured into the `argv` stanza of the kernelspec. If you take this approach, you need to apply the necessary changes to integrate with your resource manager.
-3. Determine how to interact with the resource manager's API to _discover_ the kernel and determine on which host it's running. This interaction should occur immediately following Enterprise Gateway's receipt of the kernel's connection information in its response from the kernel launcher. This extra step, performed within `confirm_remote_startup()`, is necessary to get the appropriate host name as reflected in the resource manager's API.
-4. Determine how to monitor the "job" using the resource manager API. This will become part of the `poll()` implementation to determine if the kernel is still running. This should be as quick as possible since it occurs every 3 seconds. If this is an expensive call, you may need to make some adjustments like skip the call every so often.
-5. Determine how to terminate "jobs" using the resource manager API. This will become part of the termination sequence, but probably only necessary if the message-based shutdown does not work (i.e., a last resort).
+1. Today, all invocations of kernels into resource managers use a shell or python script mechanism configured into the `argv` stanza of the kernelspec. If you take this approach, you need to apply the necessary changes to integrate with your resource manager.
+1. Determine how to interact with the resource manager's API to _discover_ the kernel and determine on which host it's running. This interaction should occur immediately following Enterprise Gateway's receipt of the kernel's connection information in its response from the kernel launcher. This extra step, performed within `confirm_remote_startup()`, is necessary to get the appropriate host name as reflected in the resource manager's API.
+1. Determine how to monitor the "job" using the resource manager API. This will become part of the `poll()` implementation to determine if the kernel is still running. This should be as quick as possible since it occurs every 3 seconds. If this is an expensive call, you may need to make some adjustments like skip the call every so often.
+1. Determine how to terminate "jobs" using the resource manager API. This will become part of the termination sequence, but probably only necessary if the message-based shutdown does not work (i.e., a last resort).
 
 ```{tip}
 Because kernel IDs are globally unique, they serve as ideal identifiers for discovering where in the cluster the kernel is running.

docs/source/developers/kernel-launcher.md

@@ -7,9 +7,9 @@ Its generally recommended that the launcher be written in the language of the ke
 To reiterate, the four tasks of a kernel launcher are:
 
 1. Create the necessary connection information based on the 5 zero-mq ports, a signature key and algorithm specifier, along with a _gateway listener_ socket.
-2. Conveyance of the connection (and listener socket) information back to the Enterprise Gateway process after encrypting the information using AES, then encrypting the AES key using the provided public key.
-3. Invocation of the target kernel.
-4. Listen for interrupt and shutdown requests from Enterprise Gateway on the communication socket and carry out the action when appropriate.
+1. Conveyance of the connection (and listener socket) information back to the Enterprise Gateway process after encrypting the information using AES, then encrypting the AES key using the provided public key.
+1. Invocation of the target kernel.
+1. Listen for interrupt and shutdown requests from Enterprise Gateway on the communication socket and carry out the action when appropriate.
 
 ## Creating the connection information
 

docs/source/operators/config-cli.md

@@ -297,11 +297,11 @@ WebhookKernelSessionManager(KernelSessionManager) options
 ---------------------------------------------------------
 --WebhookKernelSessionManager.enable_persistence=<Bool>
     Enable kernel session persistence (True or False). Default = False
-                                  (EG_KERNEL_SESSION_PERSISTENCE env var)
+    (EG_KERNEL_SESSION_PERSISTENCE env var)
     Default: False
 --WebhookKernelSessionManager.persistence_root=<Unicode>
     Identifies the root 'directory' under which the 'kernel_sessions' node will
-                                   reside.  This directory should exist.  (EG_PERSISTENCE_ROOT env var)
+    reside.  This directory should exist.  (EG_PERSISTENCE_ROOT env var)
     Default: None
 --WebhookKernelSessionManager.webhook_url=<Unicode>
     URL endpoint for webhook kernel session manager

docs/source/operators/config-dynamic.md

@@ -12,11 +12,11 @@ values, but does have the following caveats:
 1. Any configuration variables set on the command line (CLI) or via environment variables are
    NOT eligible for dynamic updates. This is because Jupyter gives those values priority over
    file-based configuration variables.
-2. Any configuration variables tied to background processing may not reflect their update if
+1. Any configuration variables tied to background processing may not reflect their update if
    the variable is not _observed_ for changes. For example, the code behind
    `RemoteKernelManager.cull_idle_timeout` may not reflect changes to the timeout period if
    that variable is not monitored (i.e., observed) for changes.
-3. Only `Configurables` registered by Enterprise Gateway are eligible for dynamic updates.
+1. Only `Configurables` registered by Enterprise Gateway are eligible for dynamic updates.
    Currently, that list consists of the following (and their subclasses): EnterpriseGatewayApp,
    RemoteKernelManager, KernelSpecManager, and KernelSessionManager.