Add Kubernetes deployment guide #899
Conversation
terrytangyuan
requested review from
ashwinb,
yanxi0830,
hardikjshah,
dltn,
raghotham,
dineshyv,
vladimirivic and
sixianyi0721
as code owners
facebook-github-bot
added
the
CLA Signed
There was a problem hiding this comment.
Hey I did something similar a few days ago: https://github.com/ellistarn/llama-operator/blob/main/manifest/deployment.yaml
| @@ -0,0 +1,192 @@ | |||
| # Kubernetes Deployment Guide | |||
There was a problem hiding this comment.
There's an open question to me as to whether or not Kubernetes maps to the Llama Stack "distribution" concept. Many of these steps would not hold true for running w/ meta-reference or ollama. This feels like a sibling of https://github.com/meta-llama/llama-stack/blob/39c34dd25f9365b09000a07de5c46dbdba27e3cb/distributions/remote-vllm/compose.yaml.
However, there is substantial overlap between the compose files for each distribution, so I think we can do better. I think we probably need to figure out how to draw the right boundaries around distributions and deployment options. In the meantime, WDYT about moving this there?
There was a problem hiding this comment.
We were discussing the same idea on Discord. My initial thought was to first provide a guide so that others can start following similar ways to deploy their selected providers to K8s (and make any necessary changes to meet their needs). Next step is to provide a packaged YAML file that's templated for K8s deployment for each provider (e.g. remote:vllm) and then we can simplify this guide. I believe we'll need a guide anyways to call out specific details.
There was a problem hiding this comment.
I'll leave this call to the maintainers :)
There was a problem hiding this comment.
@raghotham @ashwinb Any thoughts on this? I think we'll need a guide anyways. In the future, it's a matter of simplifying it once we have K8s distributions in addition to existing docker compose distributions.
|
@ellistarn Thanks for the review and suggestions! It's great to cross paths here again |
Signed-off-by: Yuan Tang <terrytangyuan@gmail.com>
Signed-off-by: Yuan Tang <terrytangyuan@gmail.com>
Signed-off-by: Yuan Tang <terrytangyuan@gmail.com>
terrytangyuan
force-pushed
the
k8s-deployment
branch
from
da127c6 to
3bcc778
Compare
There was a problem hiding this comment.
Happy to get this in for now. I agree with @ellistarn that we need to figure out how Stack distros fit in with K8S in general (I don't believe it is a clear 1:1 map either given so many things that need to happen in a real deployment and I don't know if Stack wants to assert control over them right now.)
This PR moves some content from the recent blog post to here as a more official guide for users who'd like to deploy Llama Stack on Kubernetes.