Need strategy to enable users to run MLOps workflows (how to share secrets, etc.)

[ca-scribner]

MLOps implementation requires secrets (kubeflow/other) and other services to work:

• secrets
  • acr credentials
  • aks credentials
  • service principal/secret/audience
• other services
  • mlopsbot running on kubernetes with github token that authorizes communication back to github actions/PRs

We need a way to enable users to develop in a repo that has this information.

Everyone working out of branches on a single repo is a prohibitive option. Some github actions must be on the master branch (eg: dispatch events or comment events trigger the master branch's CI). If teams used separate branches, they'd still have to commit these sorts of CI changes to the master, and could mess each other up (although with trusted users this could still work).

For secrets, might be able to use organizations so repos can inherit the secrets? Need to investigate. Innovation team or others (Jonathan Lam?) may have investigated this already.

[ca-scribner]

Blocked until Zach returns ~mid September. After that need to schedule a brainstorming session to discuss

[sylus]

@blairdrummond isn't this now unblocked with the argo fix we added so the vault secrets get injected into each persons experiments?

[blairdrummond]

Issue is more so things like CI secrets and ACR stuff. Regular users can't build images except for within a CI process in daaas-containers, and CI integrations are still a bit tricky because a semi limited number of people have the ability to add secrets (KUBE_CONFIG tokens and such). For instance, if I wanted to push a Kubeflow pipeline from a CI process from the DScD gitlab (on gitlab.com, atm), I wouldn't currently be able to do that easily.

So I think operations is 👍 inside AAW, it's the integrations outside that are trickier.

It was part of the reason I brought up the idea of a gitlab instance (either connectivity to the internal one or a separate one), or maybe another github org or project --- some kinda of umbrella where we can say:

"We have reasonable trust in the folks in this instance/org, so they will be able to create their own repos that have access to creds"

Does that kinda make sense? One analogy might be, imagine if there was a Gitlab instance with 1 group per namespace, and then that namespace had a kube-config token that would allow users to run kubeflow pipelines, configure their seldon/kfserving systems, pull/push to MinIO, etc.

Maybe @ca-scribner and I can try to flush out a more detailed list of requirements this week.

[ca-scribner]

This was meant for things like the secrets that the mlops repo needed to function, not for passing secrets to a pipeline. Like the secrets that the mlops repo has that lets the github action submit a kubeflow job, publish to our acr, etc

…

On Mon, Dec 28, 2020 at 21:27 William H ***@***.***> wrote: @blairdrummond <https://github.com/blairdrummond> isn't this now unblocked with the argo fix we added so the vault secrets get injected into each persons experiments? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#30 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALPFPIYCVL5RRKCVZK5DGF3SXE5B3ANCNFSM4QNL725A> .