Custom images / private registry

[gzt5142]

Hello,

Newby question here. . .

I'm trying to spec a nebari deployment in a sequestered environment. This will run in Azure, but with a fairly strict security posture. One of the consequences of this is that we need to vet and then copy any container images to an internal registry. As pods load their images, they won't be allowed to take from the (presumed dirty) public internet -- only from this internal registry.

My question concerns how to tell nebari-config.yaml about the now new locations for all of these images. From jupyterhub down to volume-nsf and all the rest which support various kubernetes services.

[gzt5142]

A little more background info:

I see that some of these custom images can be specified:

default_images:
  jupyterhub: "internal.registry/nebari/nebari-jupyterhub:tag"
  jupyterlab: "internal.registry/nebari/nebari-jupyterlab:tag"
  dask_worker: "internal.registry/nebari/nebari-dask-worker:tag"

That's very convenient. However, I'm not having much luck with specifying other images used across the deployment.

Example... I believe that this will set the image for conda-store:

conda_store:
  image: "custom/conda-store-server:tag"

But then what about the helm charts for minio, postgresql, redis, and volume-nfs upon which conda-store depends?

I'm getting validation errors when I try to add an overrides section to conda_store. Or is this supposed to go in extra_settings or extra_config ? Using extra_settings, the config validates -- but I don't see any files change when I render the config locally. (I suspect that I'm simply misunderstanding what render actually does.)

As I read through all the helm charts and whatnot, I see quite a few image specifications that (I think) need to be overridden... At this stage, I'm just trying to determine if that's possible, and how to do it on the nebari config side.

That's a lot of preamble for a couple of questions:

• Do I have this level of control/specificity to configure the images pulled for all corners of the deployment?
• How do I verify my changes will update in the correct helm charts? More specifically, is there an easy way to check if I found them all?

My sincere hope here is that there's an obvious concept that I've simply missed -- hoping perhaps you can straighten me out.

--gt

[rsignell]

@iameskild @aktech @viniciusdc , just for context, this is within a restricted NATO environment on Azure, where they won't allow pulling containers from non-trusted sources!

[gzt5142]

OK... so I now have most of an answer to this question.

I got a lot of help from a cloud engineer here (i.e. he did most of the hard work)... and we came to the following declarations as part of the nebari-config.yaml:

default_images:
  jupyterhub: "my_internal_registry.io/nebari/nebari-jupyterhub:2023.11.1"
  jupyterlab: "my_internal_registry.io/nebari/nebari-jupyterlab:2023.11.1"
  dask_worker: "my_internal_registry.io/nebari/nebari-dask-worker:2023.11.1"

ingress:
  terraform_overrides:
      traefik-image:
        image: "my_internal_registry.io/traefik"
        tag: "2.9.1"
security:
  keycloak:
    overrides:
      image:
        repository: "my_internal_registry.io/keycloak"
        tag: "15.0.2"
      pgchecker:
        image:
          repository: "my_internal_registry.io/busybox"
          tag: "1.36.1"
      postgresql:
        image:
          registry: "my_internal_registry.io"
          repository: "bitnami/postgresql"
          tag: "11.11.0-debian-10-r31"
jupyterhub:
  overrides:
    proxy:
      chp:
        image:
          name: "my_internal_registry.io/jupyterhub/configurable-http-proxy"
      secretSync:
        image:
          name: "my_internal_registry.io/jupyterhub/k8s-secret-sync"
          tag: "2.0.0"
    singleuser:
      image:
        name: "my_internal_registry.io/jupyterhub/k8s-singleuser-sample"
        tag: "2.0.0"
    networkTools:
      image:
        name: "my_internal_registry.io/jupyterhub/k8s-network-tools"
        tag: "2.0.0"
    scheduling:
      userScheduler:
        image:
          name: "my_internal_registry.io/kube-scheduler"
      userPlaceholder:
        image:
          name: "my_internal_registry.io/pause"
    prePuller:
      hook:
        image:
          name: "my_internal_registry.io/jupyterhub/k8s-image-awaiter"
          tag: "2.0.0"
      pause:
        image:
          name: "my_internal_registry.io/pause"
argo_workflows:
  overrides:
    controller:
      image:
        registry: "my_internal_registry.io"
        repository: "argoproj/workflow-controller"
        tag: "v3.4.4"
    executor:
      image:
        registry: "my_internal_registry.io"
        repository: "argoproj/argoexec"
        tag: "v3.4.4"
    server:
      image:
        registry: "my_internal_registry.io"
        repository: "argoproj/argocli"
        tag: "v3.4.4"
conda_store:
  image: "my_internal_registry.io/quansight/conda-store-server"
  image_tag: "2023.10.1"

I'll observer here that there's a lot of variation in how those overrides are applied -- I understand that this is likely due to the disparate sources of the various helm charts used to install those services. Related issue: deployments using third-party-sourced helm charts may use different versions/tags of some common images (busybox is a good example of this).

---

But that's only part of the story... there are several places where the nebari-config.yaml cannot be used to specify a particular image.

For example: The image used to define the nfs-server for conda-store cannot be set in nebari-config.yaml; it is hard-coded here:

nebari/src/_nebari/stages/kubernetes_services/template/modules/kubernetes/services/conda-store/worker.tf

Line 139 in efc2913

image = "gcr.io/google_containers/volume-nfs:0.8"

This image location is specified in the tf template which nebari uses to render the terraform config. Most other image locations are composed values derived from variables assigned as part of the render from nebari-config.yaml (and specified using the yaml as above).

[dharhas]

@dcmcand @costrouc ☝🏽

[costrouc]

But that's only part of the story... there are several places where the nebari-config.yaml cannot be used to specify a particular image.

@gzt5142 it is true that many of these settings/images are not overridable. One thing over time that we have been heavily working on is making sure that it is possible to override as many values as possible.

This is possibly something that needs to be improved so that all images can be pulled from trusted sources. Another possibility is our Nebari extension mechanism and overriding the entire deployment stage so that we can customize the exact image being used.

The extension route to me would be the best solution for what you are trying to do but that would not be trivial.

[gzt5142]

The extension would be a fair bit of work. . . what we did instead is a hacky short-term patch to get ourselves up and running:

We modify the installed nebari python package found under the user's site-packages folder. Each of the relevant hard-coded values was adjusted to use our custom registry. This just kicks the can down the road, as it's not really a maintainable solution. But it has got us up and going.

If I understand nebari's render command correctly, we could have also (using an unmodified nebari python package):

• nebari render -- This will create terraform inputs using the stock/standard nebari templates.
• Manually find and edit the rendered terraform in ./stages
• nebari deploy --disable-render

If that would achieve the same result, I think it a more portable strategy than what we actually did.

-gt

[dharhas]

@gzt5142 If you are able to contribute we would welcome pr's to make Nebari more configurable, also if your organization is interested in funding these features, we should talk.