Running on TrueNAS

[Renari]

I've spent most of today trying to get some form of the stable-diffusion-webui up and running on TrueNAS. To no avail, since TrueNAS apps use k8s not docker compose.

I was able to clone this, build the images, upload them to dockerhub and replicate the docker-compose config using a Truecharts custom-app. However it seems the nvidia-container-toolkit is not installed in the container.

FROM python:3.10.9-slim

I made the following changes and was able to get it up and running, this is likely the same issue others are pointing out relating to not being able to find their GPU.

diff --git a/services/AUTOMATIC1111/Dockerfile b/services/AUTOMATIC1111/Dockerfile
index b2d44d8..bb99ea7 100644
--- a/services/AUTOMATIC1111/Dockerfile
+++ b/services/AUTOMATIC1111/Dockerfile
@@ -28,15 +28,15 @@ FROM alpine:3.17 as xformers
 RUN apk add --no-cache aria2
 RUN aria2c -x 5 --dir / --out wheel.whl 'https://github.com/AbdBarho/stable-diffusion-webui-docker/releases/download/5.0.0/xformers-0.0.17.dev449-cp310-cp310-manylinux2014_x86_64.whl'
 
-FROM python:3.10.9-slim
+FROM nvidia/cuda:11.7.1-cudnn8-devel-ubuntu22.04
 
 SHELL ["/bin/bash", "-ceuxo", "pipefail"]
 
 ENV DEBIAN_FRONTEND=noninteractive PIP_PREFER_BINARY=1
 
-RUN PIP_NO_CACHE_DIR=1 pip install torch==1.13.1+cu117 torchvision --extra-index-url https://download.pytorch.org/whl/cu117
+RUN apt-get update && apt install fonts-dejavu-core rsync git jq moreutils python3 python3-dev python3-pip python3-venv git -y && apt-get clean
 
-RUN apt-get update && apt install fonts-dejavu-core rsync git jq moreutils -y && apt-get clean
+RUN PIP_NO_CACHE_DIR=1 pip install torch==1.13.1+cu117 torchvision --extra-index-url https://download.pytorch.org/whl/cu117
 
 
 RUN --mount=type=cache,target=/root/.cache/pip <<EOF
@@ -86,12 +86,10 @@ COPY . /docker
 RUN <<EOF
 python3 /docker/info.py ${ROOT}/modules/ui.py
 mv ${ROOT}/style.css ${ROOT}/user.css
-# one of the ugliest hacks I ever wrote
-sed -i 's/in_app_dir = .*/in_app_dir = True/g' /usr/local/lib/python3.10/site-packages/gradio/routes.py
 EOF
 
 WORKDIR ${ROOT}
 ENV CLI_ARGS=""
 EXPOSE 7860
 ENTRYPOINT ["/docker/entrypoint.sh"]
-CMD python -u webui.py --listen --port 7860 ${CLI_ARGS}
+CMD python3 -u webui.py --listen --port 7860 ${CLI_ARGS}

I did remove the gradio hack, since it doesn't work on the new image at that location and doesn't matter for my use case.

[AbdBarho]

If I am not mistaken, the nvidia-container-toolkit should be installed on the host, not in the container. And maybe you need to add some cli flags on the kubelet to let it see the GPU.

A quick search lead me to this:
https://kubernetes.io/docs/tasks/manage-gpus/scheduling-gpus/

this repo was not built with k8s in mind, although maybe having skaffold support would be a nice to have.

[Renari]

The above k8s documentation restricts GPU usage to a single container. The toolkit is indeed installed on the host, but the container still needs Nvidia drivers, which it did not have.

[Renari]

Turns out that gradio hack did become relevant, gradio doesn't like symlinks apparently and it seems that was intended to workaround that issue. I've added it back in, it seems the correct location with these changes is:

sed -i 's/in_app_dir = .*/in_app_dir = True/g' /usr/local/lib/python3.10/dist-packages/gradio/routes.py