Improved extensibility options

[sblair-metrostar]

During yesterday's community session we briefly discussed Metrostar's Onyx platform that's being built on Nebari and some of the complications we've run into with our initial round of integrations. We're looking to eventually be able to fully automate the setup of a new environment, including the Nebari platform, additional apps and all required configurations. Unfortunately, the current set of options provided by Nebari are pretty limited and we'd like to discuss building out something more robust.

Here are some examples of things we've attempted but didn't quite fit the mold:

• Adding the Airflow helm chart but not being able to fully automate Keycloak client creation, credential management, etc.... There's a tf_extensions feature I noticed but only seems capable of setting up a very specific type of deployment and the Keycloak client in particular hard codes the redirect url, doesn't capture or record the client secret anywhere I could find beyond passing it directly to the deployment, etc... So wasn't helpful for anything we'd want to do.
• Several apps we've added (Airflow, MLFlow, others) have data persistence needs we'd like to leverage from our cloud provider (AWS) rather than internalized but there's no good way to wire in arbitrary Terraform code that I've found. Ultimately, I'd like to be able to write a new Terraform module that would provision an S3 bucket, RDS or whatever and use parameters from those as inputs to a helm chart or something.
• No straight forward way we've found to leverage helm_extensions to reference charts in a private Github repository. On other projects I've worked this is pretty simple to do with FluxCD or ArgoCD so not sure how complicated it is using Terraform to manage everything, but would be really helpful.
• There's not much in the way of a secrets management approach baked into nebari so we need to add something, but this frequently creates a chicken or egg problem that doesn't layer in with the simple helm_extensions method that currently exists.

My initial thoughts around how to approach this would be to maybe expand a bit on the tools you already have but to mainly focus on a failsafe mechanism that could generally handle all the edge cases that won't fit the mold necessarily. Something like the following:

Improved helm_extensions:

• Expanded configuration options to enable Keycloak integration, client creation, a secret to store the connection info in cluster, etc...
• Ability to install things in other namespaces, having the one mono-namespace won't work in all cases

Custom terraform modules:

• Add support for arbitrary terraform modules that can be applied with a standard set of input variables and provider configs to the 8 you already have.
• Probably add a block to the nebari_config yaml with paths to the modules and any other relevant configuration info, like ordering or custom input variables, additional provider connections, etc...
• In my case, would need each custom module to configure provider connections for AWS, Kubernetes, Helm, and Keycloak at least, but this would obviously vary between cloud providers
• Would be really nice to be able to inject code earlier in the existing staging as well to allow for integration of secrets management solutions or other upstream dependencies that might need to precede Keycloak, Jupyter and whatnot, but I'd call this a stretch goal.

We'd be happy to build some of this out on our end and contribute it back if there's agreement on an approach.

I'm pretty sure our UI development team is also looking to extend Jupyterhub in ways I can't exactly articulate but as I understand it are difficult/impossible with the current options available. That seems like a separate discussion but wanted to put it on the radar under the umbrella of extensibility.

[dharhas]

re: UI customization, I think it is worth splitting that into a different discussion topic. Also, are you talking about JupyterHub or JupyterLab. Are we talking about JupyterHub or JupyterLab. For JupyterLab, an easy way to allow custom docker images is part of the answer.

I'll let others chime in on the other questions