python plugin system to setup/deploy/config hosts for qhub-hpc

[verhulstm]

I propose adding in a “python plugin system to setup/deploy/config hosts for qhub-hpc”. I want to be able to run something like qhub “init gcp -foundation [some plugin]”. That will use a standard python interface to call a 3rd party module. That module does whatever it does to create/deploy hosts and it returns a list of ip addresses that qhub can ssh into. Then the module returns and qhub-hpc continues. I think we can make and support 1 mock/sample module and the community can make others. @dharhas

[verhulstm]

one module could be a salt fabric, and another could be vmware vcenter for example. might be able to keep all the salt or vmware specific code out of the core codebase.

[dharhas]

This makes sense. We'd have to think through the design a bit and also think through the qhub cli commands since we will now have both a kubernetes and a hpc type deployment overloaded into one cli.

[verhulstm]

I like to use https://click.palletsprojects.com/en/7.x/

[verhulstm]

maybe implement a "style" pattern. "ghub deply --style k8s" or "qhub deploy --style hpc", and you set a default style? i assume the defualt is k8s?

[dharhas]

click is my goto cli tool as well. not sure if we are using it in qhub yet or not, I haven't been watching the codebase.

[dharhas]

style seems like the wrong word maybe fabric

[verhulstm]

i was thinking fabric would be the name for the thing that gives you the hosts

[costrouc]

I agree it would be helpful to have a plugin system becuase I REALLY don't want to limit the ways that parts can be deployed. From our earlier work on QHub Cloud it is clear to me that infrastructure and services that run on the infrastructure is a line that we need to draw to enable these pluggins. For example:

In order for QHub Cloud to be cloud agnostic we have to provision some infrastructure:

• kubernetes cluster with load balancer and something to automatically provision certs
• container registry for storing qhub containers
• shared filesystem (can be run inside kubernetes but the performance of NFS is pretty bad)

The rest are the services deployed on the kubernetes cluster

For QHub OnPrem there are things we assume about the infrastructure at the moment:

• N ubuntu 18.04/20.04 nodes exist
• you can ssh into these nodes as a user that can sudo

Similar to QHub Cloud all the services are deployed atop these machines. The nice advantage of QHub OnPrem is that the worker nodes for the cluster install minimal software/services so they can usually be tied to existing infrastructure at a company to bring them into the fleet.

[verhulstm]

I dont know how to creates cross-cloud k8s systems in a clean way. They seem noticeably not agnostic. Using TF as we use it can be made to work, but it is nasty. We might want to think about abstracting out the k8s init and control plan. It seems that more or less all could require some manual gui steps that can be pretty complex before TF can do its job.

[verhulstm]

what if we create a "qkube" project that only creates compliant opinionated k8s cluster on the main cloud providers? this could be used by others who just want a k8s cluster.

something like these possible commands? the first one uses the "qkube" project so it needs an additional option to be passed in.

init -foundation qkube -provider gcp
init -foundation gcp-hpc
init -foundation vmware-hpc

[costrouc]

We did the work of creating cross cloud Kubernetes clusters https://github.com/quansight/qhub-terraform-modules which is in Qhub Cloud. There are alternatives e.g. https://kubespray.io/#/ but still even kubespray has shortcomings that would still need terraform to create the non-kubernetes resources.

[verhulstm]

https://kubespray.io/#/ seems too complicated to me. can we be more opinionated? can we define only what we need for qhub-cloud and abstract and standardize around that across cloud vendors?

[costrouc]

Agreed. We do need to create a list of after infrastructure has been provisioned we assume X, Y, and Z.

[Adam-D-Lewis]

I'm closing old discussions in a repo cleanup effort. Feel free to re-open if needed.