Changing KERNEL_LAUNCH_TIMEOUT value for kubernetes EG

[amit-chandak-unskript]

Hi @kevin-bates , we are using EG with kubernetes kernel instances. We were using 2.6.0 version of elyra kernel and were passing the KERNEL_LAUNCH_TIMEOUT value in the kernel.json like this

{
  "language": "python",
  "display_name": "kernel (Build: VERSION)",
  "metadata": {
    "process_proxy": {
      "class_name": "enterprise_gateway.services.processproxies.k8s.KubernetesProcessProxy",
      "config": {
        "image_name": "foo:VERSION"
      }
    },
    "debugger": true
  },
  "env": {
     "KERNEL_LAUNCH_TIMEOUT": "600"
  },
  "argv": [
    "python",
    "/usr/local/share/jupyter/kernels/python_kubernetes/scripts/launch_kubernetes.py",
    "--RemoteProcessProxy.kernel-id",
    "{kernel_id}",
    "--RemoteProcessProxy.port-range",
    "{port_range}",
    "--RemoteProcessProxy.response-address",
    "{response_address}",
    "--RemoteProcessProxy.public-key",
    "{public_key}"
  ]
}

The corresponding change in kernel-pod.yaml.j2 was

# Add any custom envs here that aren't already configured for the kernel's environment
    - name: KERNEL_LAUNCH_TIMEOUT
      value: "{{ kernel_launch_timeout }}"

This used to work fine till we were on 2.6.0 elyra kernel version, but once we migrated to 3.2.0, this environment variable is not getting propagated to the kernel

ubuntu@ip-10-0-1-169:~$ kubectl describe pod jovyan-d0d77824-9f9f-43b2-9948-bd95c6f97ca4 -n eg
Name:             jovyan-d0d77824-9f9f-43b2-9948-bd95c6f97ca4
Namespace:        eg
Priority:         0
Service Account:  default
Node:             ip-10-0-1-169/10.0.1.169
Start Time:       Wed, 26 Apr 2023 12:27:03 +0000
Labels:           app=enterprise-gateway
                  component=kernel
                  kernel_id=d0d77824-9f9f-43b2-9948-bd95c6f97ca4
                  source=kernel-pod.yaml
Annotations:      cluster-autoscaler.kubernetes.io/safe-to-evict: false
Status:           Running
IP:               10.42.2.230
IPs:
  IP:  10.42.2.230
Containers:
  jovyan-d0d77824-9f9f-43b2-9948-bd95c6f97ca4:
    Container ID:   containerd://b493242eaabf9002b43b0c4ac33c5e5f87535d9846f9d65a39933c8266f3a7f4
    Image:          foo/python_kubernetes:1116
    Image ID:       foo/python_kubernetes@sha256:7a86dbec0e87dd25c49e306105dbc8742b1f1ce1a224130f628941953a4d0078
    Port:           <none>
    Host Port:      <none>
    State:          Running
      Started:      Wed, 26 Apr 2023 12:27:04 +0000
    Ready:          True
    Restart Count:  0
    Environment:
      KERNEL_LAUNCH_TIMEOUT:           40
      PATH:                            /opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
      KERNEL_USERNAME:                 jovyan
      KERNEL_GATEWAY:                  1
      KUBERNETES_SERVICE_PORT_HTTPS:   443
      KUBERNETES_SERVICE_PORT:         443
      KUBERNETES_SERVICE_HOST:         10.43.0.1
      KERNEL_POD_NAME:                 jovyan-d0d77824-9f9f-43b2-9948-bd95c6f97ca4
      KERNEL_SERVICE_ACCOUNT_NAME:     default
      KERNEL_NAMESPACE:                eg
      KERNEL_IMAGE:                    foo/python_kubernetes:1116
      KERNEL_EXECUTOR_IMAGE:           foo/python_kubernetes:1116
      KERNEL_UID:                      1000
      KERNEL_GID:                      100
      EG_MIN_PORT_RANGE_SIZE:          1000
      EG_MAX_PORT_RANGE_RETRIES:       5
      KERNEL_ID:                       d0d77824-9f9f-43b2-9948-bd95c6f97ca4
      KERNEL_LANGUAGE:                 python
      EG_IMPERSONATION_ENABLED:        False
      JPY_PARENT_PID:                  7
      LC_CTYPE:                        C.UTF-8
      PORT_RANGE:                      0..0
      PUBLIC_KEY:                      MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQCQ2obyrk6lkRLZXe+3A/vgbJtWOWy2Pbni7Ta4GDtV5SBJ/ERT63iqgSeyd1+afAZZ6z5DsvR58KVPJd3gSbe51KgqPDlvm6jEitoqTaleYu039fqIiSJXUR5KPVi4oTY6pbcMBVXcaIa/Yc1XBiqHKhqKj3YtZ/AZHjVHBH7q5QIDAQAB
      RESPONSE_ADDRESS:                10.42.2.228:8877

This is the describe pod output of the kernel launched. As you can see, the KERNEL_LAUNCH_TIMEOUT stays at default 40, instead of 600.

Please let me know if something has changed from 2.6.0 to 3.2.0 from configuration POV. Thanks in advance

[kevin-bates]

Hi @amit-chandak-unskript - thanks for opening this discussion.

Yes, it looks like there's some change in behavior wrt how the value of KERNEL_LAUNCH_TIMEOUT is propagated based on your information - thank you. I don't see anything obvious in a brief scan of the 2.6.0 and 3.2.0 code bases (but I may be looking in the wrong locations).

However, while it is true that we automatically flow KERNEL_-prefixed envs from the GatewayClient (in Lab or Notebook) to EG and from EG to the kernel, KERNEL_LAUNCH_TIMEOUT is not used within the kernel (pod) itself. Instead, like many other KERNEL_-prefixed environment variables, it is used during the kernel (pod)'s launch. That is, it is used by the process proxy running within the EG process to determine how long the process proxy should wait until it determines that the launch has timed out. As a result, configuring KERNEL_LAUNCH_TIMEOUT in the kernelspec env stanza has no influence on managing the kernel's launch timeout since it is only set into the kernel process itself.

One reason you found you needed to amend the k8s pod yaml script is due to an issue that was fixed in 3.0 via #1164 when we discovered (only on container-based process proxies) that custom names were not flowing. But, again, in the case of KERNEL_LAUNCH_TIMEOUT, its flow to the actual kernel process has no use in the base offering.

The value for KERNEL_LAUNCH_TIMEOUT is expected to be provided by the client within the env stanza of the POST /api/kernels request's body (as are all KERNEL_-prefixed envs).

I hope this helps better your understanding.

[amit-chandak-unskript]

Thanks @kevin-bates for the prompt response. Yes, it always puzzled me why having the env set helped resolved our issue, wherein the kernel would time out. It should be used by the kernel launcher (process proxy), rather than the pod itself. Thanks for clarifying that.

Now, coming back to how to change KERNEL_LAUNCH_TIMEOUT value from its default value, I didnt quite understand what you mentioned there. Isnt the env stanza in the kernel.json used during the pod's launch? If not, where should i define this environment variable to be used by the process proxy?

[kevin-bates]

When a POST on /api/kernels call comes into the EG server (i.e., a kernel start request), the message body that contains the name of the kernel also supports an env entry that is a dictionary of name/value pairs (each name is the environment variable name, and each value is its value). If you're hitting your EG server from a Notebook of Jupyter Server (Lab) server, then KERNEL_LAUNCH_TIMEOUT should reside in the env of that server process (unless you have added a means to influence the payload of the kernel start request). The gateway client software running within that server automatically transfers any KERNEL_-prefixed envs to the start request's payload when starting a kernel.

The client-side code ensures that the larger of JUPYTER_GATEWAY_REQUEST_TIMEOUT and KERNEL_LAUNCH_TIMEOUT is used for the KERNEL_LAUNCH_TIMEOUT value: https://github.com/jupyter-server/jupyter_server/blob/3ba9ac973643b56011cb6fbd63dc1093865b174e/jupyter_server/gateway/gateway_client.py#L565-L578. The former of which can be set via configuration (--GatewayClient.request_timeout or c.GatewayClient.request_timeout) or via the env.

[amit-chandak-unskript]

Thanks a lot @kevin-bates , putting the env variable in the jupyterlab server fixed the issue.