Model Storage

[tpfau]

I'm trying to use kubeai to set up a bunch of LLMs on our kubernetes cluster.
I would like to have the models stored locally on an NFS file system, which has a fast connection to the nodes, so would be suitable to be used as model storage.

I've tried setting:

cacheProfiles:
  standard-filestore:
    sharedFilesystem:
      storageClassName: "our-storage-class"

and cacheProfile: standard-filestore in the model configs, but this generates a single volume claim with 10Gigabyte, which is never enough to host the models.
I would really like to avoid having to re-downoad the models or hosting them on the individual node drives.

When I used to run vllm myself, I simply had a pvc that was mounted to /container-home/ on the vllm container and defined HF_HOME on the containers as container-home/huggingface and set VLLM_CONFIG_ROOT to /container-home/config, this allowed me to have different containers access the same model files, and avoid unnecessary storage on local machines.

Is something like this possible with kubeai and if so, is there an example for this?

[tpfau]

Ok,
I just noticed, it actually works as intended, but for some reason the pvc lists a 10Gi size, (and there are, as of now) > 20Gi in there.

[nstogner]

Added #298 to track this.

[nstogner]

this allowed me to have different containers access the same model files, and avoid unnecessary storage on local machines.

Are you still interested in taking a different mounting strategy and if so can you provide some more details on what it would solve for you over using the existing implementation?

[tpfau]

Sorry for not answering:
No, I don't think additional mounting strategies would be of any benefit for me.