Kernel-image-puller Does not get installed using Helmchart

[j-raghavan]

Hello Kevin,

I am trying to install Jupyter Enterprise gateway via Helm chart. Here is my environment that I am trying to run on.

1. OS: Ubuntu 20.04.02 64-bit
2. Microk8s (sudo snap install microk8s --classic) version installed is 1.21
3. helm 3
4. Helm chart used from https://github.com/jupyter/enterprise_gateway/tree/master/etc/kubernetes/helm/enterprise-gateway

Issue I am seeing.

ubuntu:~$ kubectl get pods -n eg
NAME                                  READY   STATUS             RESTARTS   AGE
enterprise-gateway-676b9575f7-tccmp   2/2     Running            0          5m41s
kernel-image-puller-9v8sm             0/1     CrashLoopBackOff   5          5m41s

ubuntu:~$ kubectl logs kernel-image-puller-9v8sm -n eg
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 394, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/local/lib/python3.9/http/client.py", line 1255, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/lib/python3.9/http/client.py", line 1301, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.9/http/client.py", line 1250, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.9/http/client.py", line 1010, in _send_output
    self.send(msg)
  File "/usr/local/lib/python3.9/http/client.py", line 950, in send
    self.connect()
  File "/usr/local/lib/python3.9/site-packages/docker/transport/unixconn.py", line 43, in connect
    sock.connect(self.unix_socket)
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 755, in urlopen
    retries = retries.increment(
  File "/usr/local/lib/python3.9/site-packages/urllib3/util/retry.py", line 532, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/usr/local/lib/python3.9/site-packages/urllib3/packages/six.py", line 734, in reraise
    raise value.with_traceback(tb)
  File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 394, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/local/lib/python3.9/http/client.py", line 1255, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/lib/python3.9/http/client.py", line 1301, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.9/http/client.py", line 1250, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.9/http/client.py", line 1010, in _send_output
    self.send(msg)
  File "/usr/local/lib/python3.9/http/client.py", line 950, in send
    self.connect()
  File "/usr/local/lib/python3.9/site-packages/docker/transport/unixconn.py", line 43, in connect
    sock.connect(self.unix_socket)
urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionRefusedError(111, 'Connection refused'))

Does kernelImage puller need any special permission to be set?

Values.yaml files attached.

# Enterprise Gateway image name and tag to use.
image: elyra/enterprise-gateway:2.5.0
# Enterprise Gateway image pull policy.
imagePullPolicy: IfNotPresent
# The primary port on which Enterprise Gateway is servicing requests.
port: 8888
# The  port on which Enterprise Gateway will receive kernel connection info responses.
response_port: 8877
# Update to deploy multiple replicas of EG.
replicas: 1
# Log output level.
logLevel: DEBUG
# Whether to mirror working directories.
mirrorWorkingDirs: false
# Master public IP on which to expose EG.
k8sMasterPublicIP:

kernel:
  # Kernel cluster role created by this chart.
  clusterRole: kernel-controller
  # Will start kernels in the same namespace as EG if True.
  shareGatewayNamespace: false
  # Timeout for kernel launching in seconds.
  launchTimeout: 60
  # Timeout for an idle kernel before its culled in seconds. Default is 1 hour.
  cullIdleTimeout: 3600
  # List of kernel names that are available for use.
  whitelist:
    - python_kubernetes
  # Default kernel name should be something from the whitelist
  defaultKernelName: python_kubernetes

kernelspecs:
  # Optional custom data image containing kernelspecs to use.
  image: public.ecr.aws/d3w9q8f9/kernelspecs:latest
  # Kernelspecs image pull policy.
  imagePullPolicy: Always

nfs:
  enabled: false
  # IP address of NFS server. Required if enabled.
  internalServerIPAddress:

ingress:
  enabled: false

  # Ingress resource host
  hostName: ""

  # Ingress resource annotations to be included depending in ingress controller.
  traefik:
    enabled: true
    path: /gateway
    annotations:
      kubernetes.io/ingress.class: "traefik"
      traefik.frontend.rule.type: PathPrefixStrip

  nginx:
    enabled: false
    path: /gateway/?(.*)
    annotations:
      kubernetes.io/ingress.class: "nginx"
      nginx.ingress.kubernetes.io/rewrite-target: /$1
      nginx.ingress.kubernetes.io/ssl-redirect: "false"
      nginx.ingress.kubernetes.io/force-ssl-redirect: "false"

# Kernel Image Puller (daemonset)
kip:
  enabled: true
  # Kernel Image Puller image name and tag to use.
  image: elyra/kernel-image-puller:dev
  # Kernel Image Puller image pull policy.
  imagePullPolicy: IfNotPresent
  # Determines whether the Kernel Image Puller will pull kernel images it has previously pulled
  pullPolicy: IfNotPresent
  # The interval (in seconds) at which the Kernel Image Puller fetches kernelspecs to pull kernel images.
  interval: 300

Could you please advise what may be missed here?

Best Regards,
Jayasimha

[kevin-bates]

This doesn't appear to be a manifestation of #992 but it's worth asking if your version of K8s supports docker by default.

Also, I see you're using the 2.5.0 image for EG but a 'dev' image for KIP. Do you get the same issue using the elyra/kernel-image-puller:2.5.0 image?

[amit-chandak-unskript]

Thanks @kevin-bates , yes, looks like its the same as #992, as we are using the following version of k8s.

ubuntu@ip-172-31-2-184:~$ kubectl version
Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.1", GitCommit:"632ed300f2c34f6d6d15ca4cef3d3c7073412212", GitTreeState:"clean", BuildDate:"2021-08-19T15:45:37Z", GoVersion:"go1.16.7", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"21+", GitVersion:"v1.21.3-3+90fd5f3d2aea0a", GitCommit:"90fd5f3d2aea0a5788b15a6f0a05e70381af7787", GitTreeState:"clean", BuildDate:"2021-07-16T22:01:05Z", GoVersion:"go1.16.6", Compiler:"gc", Platform:"linux/amd64"}

Docker support was pulled from 1.20 onwards as per this link.

Looking at #992 has run into some issues with containerd apis? In the absence of KIP, whats the alternate way to avoid timeouts bcos of image pull?

Thanks in advance.

[kevin-bates]

Looking at #992 has run into some issues with containerd apis? In the absence of KIP, whats the alternate way to avoid timeouts bcos of image pull?

Hi @amit-chandak-unskript - unfortunately, we have a poor story here. At this time, #995 will still launch KIP and it will detect whether it can pull images or not. If not, a warning message will be logged, followed by the necessary kernel image names as parsed from the configured kernelspecs. The logged message will instruct users to pre-seed their nodes with the listed images.

We will need to figure out how to resolve this condition, but until then, performing the parsing for the users via logging is the best we can do, unfortunately.

[amit-chandak-unskript]

Ah ok, thanks for the prompt response, unfortunately pre-seed the node is not an option for us. Will explore falling back to pre 1.20 k8 version in that case.