Cannot install MultiModeLLM on Linux Docker 2.9.5

[ac2323]

Area of Concern

• Server
• Behaviour of one or more Modules [provide name(s), e.g. ObjectDetectionYolo]
• Installer
• Runtime [e.g. Python3.7, .NET]
• Module packages [e.g. PyTorch)
• Something else

Describe the bug
MultiModeLLM fails to install

Your System (please complete the following information):

• CodeProject.AI Server version: 2.9.5
• OS: Ubuntu 22.04
• System RAM: 12.7GB
• GPU (if available): GeForce RTX 3060
• GPU RAM (if available): 12GB

See the install log below. It fails to install properly/start:

13:32:54:MultiModeLLM: Hi Docker! We will disable shared python installs for downloaded modules
13:32:54:MultiModeLLM: Installing CodeProject.AI Analysis Module
13:32:54:MultiModeLLM: ======================================================================
13:32:54:MultiModeLLM: CodeProject.AI Installer
13:32:54:MultiModeLLM: ======================================================================
13:32:54:MultiModeLLM: 15.05 GiB of 88.09 GiB available on Docker (linux ubuntu x86_64 - linux)
13:32:54:MultiModeLLM: Installing xz-utils...
13:32:56:MultiModeLLM: E: Could not get lock /var/lib/apt/lists/lock. It is held by process 23721 (apt-get)
13:32:56:MultiModeLLM: E: Unable to lock directory /var/lib/apt/lists/
13:32:57:MultiModeLLM: Reading package lists...
13:32:58:MultiModeLLM: Building dependency tree...
13:32:58:MultiModeLLM: Reading state information...
13:32:58:MultiModeLLM: 0 upgraded, 0 newly installed, 0 to remove and 47 not upgraded.
13:32:58:MultiModeLLM: General CodeProject.AI setup
13:32:58:MultiModeLLM: Setting permissions on runtimes folder...done
13:32:58:MultiModeLLM: Setting permissions on general downloads folder...done
13:32:58:MultiModeLLM: Setting permissions on module asset downloads folder...done
13:32:58:MultiModeLLM: Setting permissions on modules folder...done
13:32:58:MultiModeLLM: Setting permissions on models folder...done
13:32:58:MultiModeLLM: Setting permissions on persisted data folder...done
13:32:58:MultiModeLLM: os, name, arch = linux ubuntu (jammy) x86_64
13:32:58:MultiModeLLM: systemName, platform = Docker, linux
13:32:58:MultiModeLLM: edgeDevice =
13:32:58:MultiModeLLM: SSH = false
13:32:58:MultiModeLLM: setupMode = SetupModule
13:32:58:MultiModeLLM: executionEnvironment = Production
13:32:58:MultiModeLLM: rootDirPath = /app
13:32:58:MultiModeLLM: appRootDirPath = /app
13:32:58:MultiModeLLM: setupScriptDirPath = /app
13:32:58:MultiModeLLM: utilsScriptsDirPath = /app/scripts
13:32:58:MultiModeLLM: runtimesDirPath = /app/runtimes
13:32:58:MultiModeLLM: modulesDirPath = /app/modules
13:32:58:MultiModeLLM: externalModulesDirPath = /app/../CodeProject.AI-Modules
13:32:58:MultiModeLLM: downloadDirPath = /app/downloads
13:32:58:MultiModeLLM: downloadModuleAssetsDirPath = /app/downloads/modules/assets
13:32:58:MultiModeLLM: assetStorageUrl = https://codeproject-ai-bunny.b-cdn.net/server/assets/
13:32:58:MultiModeLLM: GPU support
13:32:58:MultiModeLLM: CUDA (NVIDIA) Present: Yes (CUDA 12.4, cuDNN 8.9.6)
13:32:58:MultiModeLLM: ROCm (AMD) Present: No
13:32:58:MultiModeLLM: MPS (Apple) Present: No
13:32:58:MultiModeLLM: Reading module settings.......done
13:32:58:MultiModeLLM: Processing module MultiModeLLM 1.1.0 (Internal)
13:32:58:MultiModeLLM: Variable Dump
13:32:58:MultiModeLLM: moduleName = MultiModeLLM
13:32:58:MultiModeLLM: moduleId = MultiModeLLM
13:32:58:MultiModeLLM: moduleVersion = 1.1.0
13:32:58:MultiModeLLM: runtime = python3.10
13:32:58:MultiModeLLM: runtimeLocation = Local
13:32:58:MultiModeLLM: installGPU = true
13:32:58:MultiModeLLM: pythonVersion = 3.10
13:32:58:MultiModeLLM: virtualEnvDirPath = /app/modules/MultiModeLLM/bin/ubuntu/python310/venv
13:32:58:MultiModeLLM: venvPythonCmdPath = /app/modules/MultiModeLLM/bin/ubuntu/python310/venv/bin/python3.10
13:32:58:MultiModeLLM: packagesDirPath = /app/modules/MultiModeLLM/bin/ubuntu/python310/venv/lib/python3.10/site-packages/
13:32:58:MultiModeLLM: moduleStartFilePath = multimode_llm_adapter.py
13:32:58:MultiModeLLM: Installing Python 3.10
13:32:58:MultiModeLLM: Python install path is /app/modules/MultiModeLLM/bin/ubuntu/python310
13:32:58:MultiModeLLM: Python 3.10 is already installed
13:33:00:MultiModeLLM: Ensuring PIP in base python install...Reading package lists...
13:33:00:MultiModeLLM: Building dependency tree...
13:33:00:MultiModeLLM: Reading state information...
13:33:00:MultiModeLLM: 0 upgraded, 0 newly installed, 1 reinstalled, 0 to remove and 47 not upgraded.
13:33:00:MultiModeLLM: Need to get 139 kB of archives.
13:33:00:MultiModeLLM: After this operation, 0 B of additional disk space will be used.
13:33:00:MultiModeLLM: Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-distutils all 3.10.8-122.04 [139 kB]
13:33:01:MultiModeLLM: debconf: delaying package configuration, since apt-utils is not installed
13:33:01:MultiModeLLM: Fetched 139 kB in 0s (1151 kB/s)
13:33:01:MultiModeLLM: (Reading database ...
13:33:01:MultiModeLLM: (Reading database ... 5%
13:33:01:MultiModeLLM: (Reading database ... 10%
13:33:01:MultiModeLLM: (Reading database ... 15%
13:33:01:MultiModeLLM: (Reading database ... 20%
13:33:01:MultiModeLLM: (Reading database ... 25%
13:33:01:MultiModeLLM: (Reading database ... 30%
13:33:01:MultiModeLLM: (Reading database ... 35%
13:33:01:MultiModeLLM: (Reading database ... 40%
13:33:01:MultiModeLLM: (Reading database ... 45%
13:33:01:MultiModeLLM: (Reading database ... 50%
13:33:01:MultiModeLLM: (Reading database ... 55%
13:33:01:MultiModeLLM: (Reading database ... 60%
13:33:01:MultiModeLLM: (Reading database ... 65%
13:33:01:MultiModeLLM: (Reading database ... 70%
13:33:01:MultiModeLLM: (Reading database ... 75%
13:33:01:MultiModeLLM: (Reading database ... 80%
13:33:01:MultiModeLLM: (Reading database ... 85%
13:33:01:MultiModeLLM: (Reading database ... 90%
13:33:01:MultiModeLLM: (Reading database ... 95%
13:33:01:MultiModeLLM: (Reading database ... 100%
13:33:01:MultiModeLLM: (Reading database ... 31194 files and directories currently installed.)
13:33:01:MultiModeLLM: Preparing to unpack .../python3-distutils_3.10.8-122.04_all.deb ...
13:33:01:MultiModeLLM: Unpacking python3-distutils (3.10.8-122.04) over (3.10.8-122.04) ...
13:33:01:MultiModeLLM: Setting up python3-distutils (3.10.8-122.04) ...
13:33:02:MultiModeLLM: /usr/bin/python3.10: No module named ensurepip
13:33:02:MultiModeLLM: done
13:33:03:MultiModeLLM: Upgrading PIP in base python install...Requirement already satisfied: pip in /usr/lib/python3/dist-packages (22.0.2)
13:33:03:MultiModeLLM: Collecting pip
13:33:03:MultiModeLLM: Using cached pip-24.3.1-py3-none-any.whl (1.8 MB)
13:33:03:MultiModeLLM: Installing collected packages: pip
13:33:03:MultiModeLLM: Attempting uninstall: pip
13:33:03:MultiModeLLM: Found existing installation: pip 22.0.2
13:33:03:MultiModeLLM: Not uninstalling pip at /usr/lib/python3/dist-packages, outside environment /usr
13:33:03:MultiModeLLM: Can't uninstall 'pip'. No files were found to uninstall.
13:33:04:MultiModeLLM: Successfully installed pip-24.3.1
13:33:04:MultiModeLLM: WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
13:33:04:MultiModeLLM: done
13:33:04:MultiModeLLM: Installing Virtual Environment tools for Linux...
13:33:04:MultiModeLLM: Searching for installed dependencies:
13:33:09:MultiModeLLM: -> python3-pip python3-setuptools python3.10-venv done
13:33:09:MultiModeLLM: Installing missing dependencies:
13:33:09:MultiModeLLM: -> python3.10-venv
13:33:09:MultiModeLLM: Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease
13:33:09:MultiModeLLM: Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease
13:33:09:MultiModeLLM: Hit:3 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64 InRelease
13:33:09:MultiModeLLM: Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease
13:33:09:MultiModeLLM: Hit:5 http://security.ubuntu.com/ubuntu jammy-security InRelease
13:33:09:MultiModeLLM: Hit:6 https://ppa.launchpadcontent.net/deadsnakes/ppa/ubuntu jammy InRelease
13:33:09:MultiModeLLM: Hit:7 https://ppa.launchpadcontent.net/dotnet/backports/ubuntu jammy InRelease
13:33:11:MultiModeLLM: Reading package lists...
13:33:12:MultiModeLLM: Reading package lists...
13:33:12:MultiModeLLM: Building dependency tree...
13:33:12:MultiModeLLM: Reading state information...
13:33:12:MultiModeLLM: The following additional packages will be installed:
13:33:12:MultiModeLLM: python3-pip-whl python3-setuptools-whl
13:33:12:MultiModeLLM: The following NEW packages will be installed:
13:33:12:MultiModeLLM: python3-pip-whl python3-setuptools-whl python3.10-venv
13:33:12:MultiModeLLM: 0 upgraded, 3 newly installed, 0 to remove and 47 not upgraded.
13:33:12:MultiModeLLM: Need to get 2474 kB of archives.
13:33:12:MultiModeLLM: After this operation, 2885 kB of additional disk space will be used.
13:33:12:MultiModeLLM: Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 python3-pip-whl all 22.0.2+dfsg-1ubuntu0.5 [1680 kB]
13:33:13:MultiModeLLM: Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 python3-setuptools-whl all 59.6.0-1.2ubuntu0.22.04.2 [788 kB]
13:33:13:MultiModeLLM: Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 python3.10-venv amd64 3.10.12-122.04.7 [5718 B]
13:33:13:MultiModeLLM: debconf: delaying package configuration, since apt-utils is not installed
13:33:13:MultiModeLLM: Fetched 2474 kB in 0s (10.4 MB/s)
13:33:13:MultiModeLLM: Selecting previously unselected package python3-pip-whl.
13:33:13:MultiModeLLM: (Reading database ...
13:33:13:MultiModeLLM: (Reading database ... 5%
13:33:13:MultiModeLLM: (Reading database ... 10%
13:33:13:MultiModeLLM: (Reading database ... 15%
13:33:13:MultiModeLLM: (Reading database ... 20%
13:33:13:MultiModeLLM: (Reading database ... 25%
13:33:13:MultiModeLLM: (Reading database ... 30%
13:33:13:MultiModeLLM: (Reading database ... 35%
13:33:13:MultiModeLLM: (Reading database ... 40%
13:33:13:MultiModeLLM: (Reading database ... 45%
13:33:13:MultiModeLLM: (Reading database ... 50%
13:33:13:MultiModeLLM: (Reading database ... 55%
13:33:13:MultiModeLLM: (Reading database ... 60%
13:33:13:MultiModeLLM: (Reading database ... 65%
13:33:13:MultiModeLLM: (Reading database ... 70%
13:33:13:MultiModeLLM: (Reading database ... 75%
13:33:13:MultiModeLLM: (Reading database ... 80%
13:33:13:MultiModeLLM: (Reading database ... 85%
13:33:13:MultiModeLLM: (Reading database ... 90%
13:33:13:MultiModeLLM: (Reading database ... 95%
13:33:13:MultiModeLLM: (Reading database ... 100%
13:33:13:MultiModeLLM: (Reading database ... 31194 files and directories currently installed.)
13:33:13:MultiModeLLM: Preparing to unpack .../python3-pip-whl_22.0.2+dfsg-1ubuntu0.5_all.deb ...
13:33:13:MultiModeLLM: Unpacking python3-pip-whl (22.0.2+dfsg-1ubuntu0.5) ...
13:33:13:MultiModeLLM: Selecting previously unselected package python3-setuptools-whl.
13:33:13:MultiModeLLM: Preparing to unpack .../python3-setuptools-whl_59.6.0-1.2ubuntu0.22.04.2_all.deb ...
13:33:13:MultiModeLLM: Unpacking python3-setuptools-whl (59.6.0-1.2ubuntu0.22.04.2) ...
13:33:13:MultiModeLLM: Selecting previously unselected package python3.10-venv.
13:33:13:MultiModeLLM: Preparing to unpack .../python3.10-venv_3.10.12-122.04.7_amd64.deb ...
13:33:13:MultiModeLLM: Unpacking python3.10-venv (3.10.12-122.04.7) ...
13:33:13:MultiModeLLM: Setting up python3-setuptools-whl (59.6.0-1.2ubuntu0.22.04.2) ...
13:33:13:MultiModeLLM: Setting up python3-pip-whl (22.0.2+dfsg-1ubuntu0.5) ...
13:33:13:MultiModeLLM: Setting up python3.10-venv (3.10.12-1~22.04.7) ...
13:33:13:MultiModeLLM: Creating Virtual Environment (Local)...Install path is /app/modules/MultiModeLLM/bin/ubuntu/python310
13:33:17:MultiModeLLM: done
13:33:17:MultiModeLLM: Checking for Python 3.10...(Found Python 3.10.12) All good
13:33:20:MultiModeLLM: Upgrading PIP in virtual environment... done
13:33:22:MultiModeLLM: Installing updated setuptools in venv... done
13:33:23:Response timeout. Try increasing the timeout value
13:33:23:MultiModeLLM: Reading package lists...
13:33:24:MultiModeLLM: Building dependency tree...
13:33:24:MultiModeLLM: Reading state information...
13:33:24:MultiModeLLM: The following additional packages will be installed:
13:33:24:MultiModeLLM: git git-man less libcbor0.8 liberror-perl libfido2-1 openssh-client xauth
13:33:24:MultiModeLLM: Suggested packages:
13:33:24:MultiModeLLM: gettext-base git-daemon-run | git-daemon-sysvinit git-doc git-email git-gui
13:33:24:MultiModeLLM: gitk gitweb git-cvs git-mediawiki git-svn keychain libpam-ssh monkeysphere
13:33:24:MultiModeLLM: ssh-askpass
13:33:24:MultiModeLLM: The following NEW packages will be installed:
13:33:24:MultiModeLLM: git git-lfs git-man less libcbor0.8 liberror-perl libfido2-1 openssh-client
13:33:24:MultiModeLLM: xauth
13:33:24:MultiModeLLM: 0 upgraded, 9 newly installed, 0 to remove and 47 not upgraded.
13:33:24:MultiModeLLM: Need to get 8873 kB of archives.
13:33:24:MultiModeLLM: After this operation, 35.4 MB of additional disk space will be used.
13:33:24:MultiModeLLM: Do you want to continue? [Y/n] Abort.
13:33:24:MultiModeLLM: Looking for model: microsoft/Phi-3-vision-128k-instruct-onnx-cuda in cuda-int4-rtn-block-32...downloading...Ensuring PIP is installed and up to date...
13:33:24:MultiModeLLM: Searching for installed dependencies:
13:33:25:MultiModeLLM: -> python3-pip done
13:33:25:MultiModeLLM: All dependencies already installed.
13:33:25:MultiModeLLM: Ensuring PIP compatibility ...
13:33:27:MultiModeLLM: Looking in links: /tmp/tmpjl5rkejc
13:33:27:MultiModeLLM: Requirement already satisfied: setuptools in ./bin/ubuntu/python310/venv/lib/python3.10/site-packages (75.6.0)
13:33:27:MultiModeLLM: Requirement already satisfied: pip in ./bin/ubuntu/python310/venv/lib/python3.10/site-packages (24.3.1)
13:33:28:MultiModeLLM: Installing huggingface-hub...Processing huggingface-hub...Installing huggingface-hub (huggingface-hub[cli])...Collecting huggingface-hub[cli]
13:33:28:MultiModeLLM: Downloading huggingface_hub-0.26.5-py3-none-any.whl.metadata (13 kB)
13:33:28:MultiModeLLM: Collecting filelock (from huggingface-hub[cli])
13:33:28:MultiModeLLM: Using cached filelock-3.16.1-py3-none-any.whl.metadata (2.9 kB)
13:33:29:MultiModeLLM: Collecting fsspec>=2023.5.0 (from huggingface-hub[cli])
13:33:29:MultiModeLLM: Using cached fsspec-2024.10.0-py3-none-any.whl.metadata (11 kB)
13:33:29:MultiModeLLM: Collecting packaging>=20.9 (from huggingface-hub[cli])
13:33:29:MultiModeLLM: Using cached packaging-24.2-py3-none-any.whl.metadata (3.2 kB)
13:33:29:MultiModeLLM: Collecting pyyaml>=5.1 (from huggingface-hub[cli])
13:33:29:MultiModeLLM: Downloading PyYAML-6.0.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (2.1 kB)
13:33:29:MultiModeLLM: Collecting requests (from huggingface-hub[cli])
13:33:29:MultiModeLLM: Using cached requests-2.32.3-py3-none-any.whl.metadata (4.6 kB)
13:33:29:MultiModeLLM: Collecting tqdm>=4.42.1 (from huggingface-hub[cli])
13:33:29:MultiModeLLM: Using cached tqdm-4.67.1-py3-none-any.whl.metadata (57 kB)
13:33:29:MultiModeLLM: Collecting typing-extensions>=3.7.4.3 (from huggingface-hub[cli])
13:33:29:MultiModeLLM: Using cached typing_extensions-4.12.2-py3-none-any.whl.metadata (3.0 kB)
13:33:29:MultiModeLLM: Collecting InquirerPy==0.3.4 (from huggingface-hub[cli])
13:33:29:MultiModeLLM: Downloading InquirerPy-0.3.4-py3-none-any.whl.metadata (8.1 kB)
13:33:29:MultiModeLLM: Collecting pfzy<0.4.0,>=0.3.1 (from InquirerPy==0.3.4->huggingface-hub[cli])
13:33:29:MultiModeLLM: Downloading pfzy-0.3.4-py3-none-any.whl.metadata (4.9 kB)
13:33:29:MultiModeLLM: Collecting prompt-toolkit<4.0.0,>=3.0.1 (from InquirerPy==0.3.4->huggingface-hub[cli])
13:33:29:MultiModeLLM: Downloading prompt_toolkit-3.0.48-py3-none-any.whl.metadata (6.4 kB)
13:33:29:MultiModeLLM: Collecting charset-normalizer<4,>=2 (from requests->huggingface-hub[cli])
13:33:29:MultiModeLLM: Downloading charset_normalizer-3.4.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (34 kB)
13:33:29:MultiModeLLM: Collecting idna<4,>=2.5 (from requests->huggingface-hub[cli])
13:33:29:MultiModeLLM: Using cached idna-3.10-py3-none-any.whl.metadata (10 kB)
13:33:30:MultiModeLLM: Collecting urllib3<3,>=1.21.1 (from requests->huggingface-hub[cli])
13:33:30:MultiModeLLM: Using cached urllib3-2.2.3-py3-none-any.whl.metadata (6.5 kB)
13:33:30:MultiModeLLM: Collecting certifi>=2017.4.17 (from requests->huggingface-hub[cli])
13:33:30:MultiModeLLM: Using cached certifi-2024.8.30-py3-none-any.whl.metadata (2.2 kB)
13:33:30:MultiModeLLM: Collecting wcwidth (from prompt-toolkit<4.0.0,>=3.0.1->InquirerPy==0.3.4->huggingface-hub[cli])
13:33:30:MultiModeLLM: Downloading wcwidth-0.2.13-py2.py3-none-any.whl.metadata (14 kB)
13:33:30:MultiModeLLM: Downloading InquirerPy-0.3.4-py3-none-any.whl (67 kB)
13:33:30:MultiModeLLM: Using cached fsspec-2024.10.0-py3-none-any.whl (179 kB)
13:33:30:MultiModeLLM: Using cached packaging-24.2-py3-none-any.whl (65 kB)
13:33:30:MultiModeLLM: Downloading PyYAML-6.0.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (751 kB)
13:33:30:MultiModeLLM: ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 751.2/751.2 kB 16.2 MB/s eta 0:00:00
13:33:30:MultiModeLLM: Using cached tqdm-4.67.1-py3-none-any.whl (78 kB)
13:33:30:MultiModeLLM: Using cached typing_extensions-4.12.2-py3-none-any.whl (37 kB)
13:33:30:MultiModeLLM: Using cached filelock-3.16.1-py3-none-any.whl (16 kB)
13:33:30:MultiModeLLM: Downloading huggingface_hub-0.26.5-py3-none-any.whl (447 kB)
13:33:30:MultiModeLLM: Using cached requests-2.32.3-py3-none-any.whl (64 kB)
13:33:30:MultiModeLLM: Using cached certifi-2024.8.30-py3-none-any.whl (167 kB)
13:33:30:MultiModeLLM: Downloading charset_normalizer-3.4.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (144 kB)
13:33:30:MultiModeLLM: Using cached idna-3.10-py3-none-any.whl (70 kB)
13:33:30:MultiModeLLM: Downloading pfzy-0.3.4-py3-none-any.whl (8.5 kB)
13:33:30:MultiModeLLM: Downloading prompt_toolkit-3.0.48-py3-none-any.whl (386 kB)
13:33:30:MultiModeLLM: Using cached urllib3-2.2.3-py3-none-any.whl (126 kB)
13:33:30:MultiModeLLM: Downloading wcwidth-0.2.13-py2.py3-none-any.whl (34 kB)
13:33:30:MultiModeLLM: Installing collected packages: wcwidth, urllib3, typing-extensions, tqdm, pyyaml, prompt-toolkit, pfzy, packaging, idna, fsspec, filelock, charset-normalizer, certifi, requests, InquirerPy, huggingface-hub
13:33:31:MultiModeLLM: Successfully installed InquirerPy-0.3.4 certifi-2024.8.30 charset-normalizer-3.4.0 filelock-3.16.1 fsspec-2024.10.0 huggingface-hub-0.26.5 idna-3.10 packaging-24.2 pfzy-0.3.4 prompt-toolkit-3.0.48 pyyaml-6.0.2 requests-2.32.3 tqdm-4.67.1 typing-extensions-4.12.2 urllib3-2.2.3 wcwidth-0.2.13
13:33:32:MultiModeLLM: (✅ checked) done
13:33:33:MultiModeLLM: Fetching 11 files: 0%| | 0/11 [00:00
13:33:33:MultiModeLLM: Downloading 'cuda-int4-rtn-block-32/genai_config.json' to '/app/modules/MultiModeLLM/.cache/huggingface/download/cuda-int4-rtn-block-32/genai_config.json.97520d2c308ac600466044218d965ea1a9771777.incomplete'
13:33:33:MultiModeLLM: Downloading 'cuda-int4-rtn-block-32/phi-3-v-128k-instruct-text-embedding.onnx' to '/app/modules/MultiModeLLM/.cache/huggingface/download/cuda-int4-rtn-block-32/phi-3-v-128k-instruct-text-embedding.onnx.1c7c1cfa5b18a03f0334b02bf774d0e069472ae110aaa786a3b2cc8f37ad15fa.incomplete'
13:33:33:MultiModeLLM: Downloading 'cuda-int4-rtn-block-32/processor_config.json' to '/app/modules/MultiModeLLM/.cache/huggingface/download/cuda-int4-rtn-block-32/processor_config.json.c217a60033b4cfbfed7dac284912ac92744470f5.incomplete'
13:33:33:MultiModeLLM: Downloading 'cuda-int4-rtn-block-32/phi-3-v-128k-instruct-vision.onnx' to '/app/modules/MultiModeLLM/.cache/huggingface/download/cuda-int4-rtn-block-32/phi-3-v-128k-instruct-vision.onnx.339fc588156547028e199b16294c95d9b8e86a7b2178de8e54bfa83f2984ece6.incomplete'
13:33:33:MultiModeLLM: Downloading 'cuda-int4-rtn-block-32/phi-3-v-128k-instruct-text.onnx' to '/app/modules/MultiModeLLM/.cache/huggingface/download/cuda-int4-rtn-block-32/phi-3-v-128k-instruct-text.onnx.7dddf37fac48112a8bd8052e3d3cddb8c7b148f805c559509db99d8ddd1b191f.incomplete'
13:33:33:MultiModeLLM: Downloading 'cuda-int4-rtn-block-32/phi-3-v-128k-instruct-text-embedding.onnx.data' to '/app/modules/MultiModeLLM/.cache/huggingface/download/cuda-int4-rtn-block-32/phi-3-v-128k-instruct-text-embedding.onnx.data.bbb5071dffdeee95cf3edda83cbd651f066fea4d13f537f3287d0abb4af20e1c.incomplete'
13:33:33:MultiModeLLM: Downloading 'cuda-int4-rtn-block-32/phi-3-v-128k-instruct-text.onnx.data' to '/app/modules/MultiModeLLM/.cache/huggingface/download/cuda-int4-rtn-block-32/phi-3-v-128k-instruct-text.onnx.data.9bc8568a35a27a07a0801045cbaf5feeb9a2a1c85af745600ea6345cd1c2b60c.incomplete'
13:33:33:MultiModeLLM: Download complete. Moving file to /app/modules/MultiModeLLM/cuda-int4-rtn-block-32/genai_config.json
13:33:33:MultiModeLLM: Fetching 11 files: 9%|▉ | 1/11 [00:00<00:03, 2.57it/s]Download complete. Moving file to /app/modules/MultiModeLLM/cuda-int4-rtn-block-32/processor_config.json
13:33:33:MultiModeLLM: Downloading 'cuda-int4-rtn-block-32/special_tokens_map.json' to '/app/modules/MultiModeLLM/.cache/huggingface/download/cuda-int4-rtn-block-32/special_tokens_map.json.f3809994ef386c0d3d822fb34f2ed4a2d179889c.incomplete'
13:33:34:MultiModeLLM: Downloading 'cuda-int4-rtn-block-32/tokenizer.json' to '/app/modules/MultiModeLLM/.cache/huggingface/download/cuda-int4-rtn-block-32/tokenizer.json.93333dd684b3068e287195d8312cb6e00bca311a.incomplete'
13:33:34:MultiModeLLM: Download complete. Moving file to /app/modules/MultiModeLLM/cuda-int4-rtn-block-32/special_tokens_map.json
13:33:34:MultiModeLLM: Downloading 'cuda-int4-rtn-block-32/tokenizer_config.json' to '/app/modules/MultiModeLLM/.cache/huggingface/download/cuda-int4-rtn-block-32/tokenizer_config.json.995bec9e081c0c5170cd4fed4356a834ef7ca875.incomplete'
13:33:34:MultiModeLLM: Download complete. Moving file to /app/modules/MultiModeLLM/cuda-int4-rtn-block-32/phi-3-v-128k-instruct-vision.onnx
13:33:34:MultiModeLLM: Download complete. Moving file to /app/modules/MultiModeLLM/cuda-int4-rtn-block-32/tokenizer_config.json
13:33:34:MultiModeLLM: Download complete. Moving file to /app/modules/MultiModeLLM/cuda-int4-rtn-block-32/phi-3-v-128k-instruct-text-embedding.onnx
13:33:34:MultiModeLLM: Fetching 11 files: 18%|█▊ | 2/11 [00:00<00:03, 2.50it/s]Download complete. Moving file to /app/modules/MultiModeLLM/cuda-int4-rtn-block-32/tokenizer.json
13:33:35:MultiModeLLM: Download complete. Moving file to /app/modules/MultiModeLLM/cuda-int4-rtn-block-32/phi-3-v-128k-instruct-text.onnx
13:33:41:MultiModeLLM: Download complete. Moving file to /app/modules/MultiModeLLM/cuda-int4-rtn-block-32/phi-3-v-128k-instruct-text-embedding.onnx.data
13:33:51:MultiModeLLM: Fetching 11 files: 27%|██▋ | 3/11 [00:07<00:27, 3.43s/it]Download complete. Moving file to /app/modules/MultiModeLLM/cuda-int4-rtn-block-32/phi-3-v-128k-instruct-vision.onnx.data
13:34:28:MultiModeLLM: Download complete. Moving file to /app/modules/MultiModeLLM/cuda-int4-rtn-block-32/phi-3-v-128k-instruct-text.onnx.data
13:34:28:MultiModeLLM: Fetching 11 files: 45%|████▌ | 5/11 [00:54<01:28, 14.81s/it]
13:34:28:MultiModeLLM: Fetching 11 files: 100%|██████████| 11/11 [00:54<00:00, 4.99s/it]
13:34:28:MultiModeLLM: /app/modules/MultiModeLLM
13:34:28:MultiModeLLM: Done.
13:34:28:MultiModeLLM: Installing Python packages for MultiModeLLM
13:34:28:MultiModeLLM: Installing GPU-enabled libraries: If available
13:34:28:MultiModeLLM: Ensuring PIP is installed and up to date...
13:34:28:MultiModeLLM: Searching for installed dependencies:
13:34:30:MultiModeLLM: -> python3-pip done
13:34:30:MultiModeLLM: All dependencies already installed.
13:34:30:MultiModeLLM: Ensuring PIP compatibility ...
13:34:31:MultiModeLLM: Looking in links: /tmp/tmp33a8c40t
13:34:31:MultiModeLLM: Requirement already satisfied: setuptools in ./bin/ubuntu/python310/venv/lib/python3.10/site-packages (75.6.0)
13:34:31:MultiModeLLM: Requirement already satisfied: pip in ./bin/ubuntu/python310/venv/lib/python3.10/site-packages (24.3.1)
13:34:32:MultiModeLLM: CUDA version is 12.4 (12_4 / 12)
13:34:32:MultiModeLLM: Python packages will be specified by requirements.cuda12.txt
13:34:33:MultiModeLLM: - Installing NumPy, a package for scientific computing...Checking ...Check done...Installing numpy...Collecting numpy
13:34:33:MultiModeLLM: Downloading numpy-2.2.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (62 kB)
13:34:33:MultiModeLLM: Downloading numpy-2.2.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (16.4 MB)
13:34:33:MultiModeLLM: ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 16.4/16.4 MB 40.5 MB/s eta 0:00:00
13:34:34:MultiModeLLM: Installing collected packages: numpy
13:34:36:MultiModeLLM: Successfully installed numpy-2.2.0
13:34:36:MultiModeLLM: WARNING: Target directory /app/modules/MultiModeLLM/bin/ubuntu/python310/venv/lib/python3.10/site-packages/bin already exists. Specify --upgrade to force replacement.
13:34:37:MultiModeLLM: (✅ checked) done
13:34:38:MultiModeLLM: - Installing onnxruntime-genai for CUDA, the ONNX Runtime generate() API...Checking ...Check done...Installing onnxruntime-genai-cuda...Looking in indexes: https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/onnxruntime-cuda-12/pypi/simple/
13:34:38:MultiModeLLM: Collecting onnxruntime-genai-cuda
13:34:38:MultiModeLLM: Downloading https://aiinfra.pkgs.visualstudio.com/2692857e-05ef-43b4-ba9c-ccf1c22c437c/_packaging/9387c3aa-d9ad-4513-968c-383f6f7f53b8/pypi/download/onnxruntime-genai-cuda/0.4/onnxruntime_genai_cuda-0.4.0-cp310-cp310-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl (15.3 MB)
13:34:40:MultiModeLLM: ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 15.3/15.3 MB 12.7 MB/s eta 0:00:00
13:34:40:MultiModeLLM: Collecting numpy>=1.21.6 (from onnxruntime-genai-cuda)
13:34:40:MultiModeLLM: Downloading https://aiinfra.pkgs.visualstudio.com/2692857e-05ef-43b4-ba9c-ccf1c22c437c/_packaging/9387c3aa-d9ad-4513-968c-383f6f7f53b8/pypi/download/numpy/2.1.2/numpy-2.1.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (16.3 MB)
13:34:40:MultiModeLLM: ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 16.3/16.3 MB 42.8 MB/s eta 0:00:00
13:34:41:MultiModeLLM: Collecting onnxruntime-gpu>=1.19.0 (from onnxruntime-genai-cuda)
13:34:41:MultiModeLLM: Downloading https://aiinfra.pkgs.visualstudio.com/2692857e-05ef-43b4-ba9c-ccf1c22c437c/_packaging/9387c3aa-d9ad-4513-968c-383f6f7f53b8/pypi/download/onnxruntime-gpu/1.19.2/onnxruntime_gpu-1.19.2-cp310-cp310-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl (226.2 MB)
13:34:58:MultiModeLLM: ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 226.2/226.2 MB 13.7 MB/s eta 0:00:00
13:34:58:MultiModeLLM: Collecting coloredlogs (from onnxruntime-gpu>=1.19.0->onnxruntime-genai-cuda)
13:34:58:MultiModeLLM: Downloading https://aiinfra.pkgs.visualstudio.com/2692857e-05ef-43b4-ba9c-ccf1c22c437c/_packaging/9387c3aa-d9ad-4513-968c-383f6f7f53b8/pypi/download/coloredlogs/15.0.1/coloredlogs-15.0.1-py2.py3-none-any.whl (46 kB)
13:34:58:MultiModeLLM: Collecting flatbuffers (from onnxruntime-gpu>=1.19.0->onnxruntime-genai-cuda)
13:34:59:MultiModeLLM: Downloading https://aiinfra.pkgs.visualstudio.com/2692857e-05ef-43b4-ba9c-ccf1c22c437c/_packaging/9387c3aa-d9ad-4513-968c-383f6f7f53b8/pypi/download/flatbuffers/24.3.25/flatbuffers-24.3.25-py2.py3-none-any.whl (26 kB)
13:34:59:MultiModeLLM: Collecting packaging (from onnxruntime-gpu>=1.19.0->onnxruntime-genai-cuda)
13:34:59:MultiModeLLM: Downloading https://aiinfra.pkgs.visualstudio.com/2692857e-05ef-43b4-ba9c-ccf1c22c437c/_packaging/9387c3aa-d9ad-4513-968c-383f6f7f53b8/pypi/download/packaging/24.1/packaging-24.1-py3-none-any.whl (53 kB)
13:34:59:MultiModeLLM: Collecting protobuf (from onnxruntime-gpu>=1.19.0->onnxruntime-genai-cuda)
13:34:59:MultiModeLLM: Downloading https://aiinfra.pkgs.visualstudio.com/2692857e-05ef-43b4-ba9c-ccf1c22c437c/_packaging/9387c3aa-d9ad-4513-968c-383f6f7f53b8/pypi/download/protobuf/5.28.3/protobuf-5.28.3-cp38-abi3-manylinux2014_x86_64.whl (316 kB)
13:34:59:MultiModeLLM: Collecting sympy (from onnxruntime-gpu>=1.19.0->onnxruntime-genai-cuda)
13:35:00:MultiModeLLM: Downloading https://aiinfra.pkgs.visualstudio.com/2692857e-05ef-43b4-ba9c-ccf1c22c437c/_packaging/9387c3aa-d9ad-4513-968c-383f6f7f53b8/pypi/download/sympy/1.13.3/sympy-1.13.3-py3-none-any.whl (6.2 MB)
13:35:00:MultiModeLLM: ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 6.2/6.2 MB 20.9 MB/s eta 0:00:00
13:35:00:MultiModeLLM: Collecting humanfriendly>=9.1 (from coloredlogs->onnxruntime-gpu>=1.19.0->onnxruntime-genai-cuda)
13:35:00:MultiModeLLM: Downloading https://aiinfra.pkgs.visualstudio.com/2692857e-05ef-43b4-ba9c-ccf1c22c437c/_packaging/9387c3aa-d9ad-4513-968c-383f6f7f53b8/pypi/download/humanfriendly/10/humanfriendly-10.0-py2.py3-none-any.whl (86 kB)
13:35:00:MultiModeLLM: Collecting mpmath<1.4,>=1.1.0 (from sympy->onnxruntime-gpu>=1.19.0->onnxruntime-genai-cuda)
13:35:01:MultiModeLLM: Downloading https://aiinfra.pkgs.visualstudio.com/2692857e-05ef-43b4-ba9c-ccf1c22c437c/_packaging/9387c3aa-d9ad-4513-968c-383f6f7f53b8/pypi/download/mpmath/1.3/mpmath-1.3.0-py3-none-any.whl (536 kB)
13:35:01:MultiModeLLM: ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 536.2/536.2 kB 6.7 MB/s eta 0:00:00
13:35:01:MultiModeLLM: Installing collected packages: mpmath, flatbuffers, sympy, protobuf, packaging, numpy, humanfriendly, coloredlogs, onnxruntime-gpu, onnxruntime-genai-cuda
13:35:14:MultiModeLLM: Successfully installed coloredlogs-15.0.1 flatbuffers-24.3.25 humanfriendly-10.0 mpmath-1.3.0 numpy-2.1.2 onnxruntime-genai-cuda-0.4.0 onnxruntime-gpu-1.19.2 packaging-24.1 protobuf-5.28.3 sympy-1.13.3
13:35:18:MultiModeLLM: WARNING: Target directory /app/modules/MultiModeLLM/bin/ubuntu/python310/venv/lib/python3.10/site-packages/pycache already exists. Specify --upgrade to force replacement.
13:35:18:MultiModeLLM: WARNING: Target directory /app/modules/MultiModeLLM/bin/ubuntu/python310/venv/lib/python3.10/site-packages/numpy.libs already exists. Specify --upgrade to force replacement.
13:35:18:MultiModeLLM: WARNING: Target directory /app/modules/MultiModeLLM/bin/ubuntu/python310/venv/lib/python3.10/site-packages/packaging already exists. Specify --upgrade to force replacement.
13:35:18:MultiModeLLM: WARNING: Target directory /app/modules/MultiModeLLM/bin/ubuntu/python310/venv/lib/python3.10/site-packages/numpy already exists. Specify --upgrade to force replacement.
13:35:18:MultiModeLLM: WARNING: Target directory /app/modules/MultiModeLLM/bin/ubuntu/python310/venv/lib/python3.10/site-packages/bin already exists. Specify --upgrade to force replacement.
13:35:19:MultiModeLLM: (✅ checked) done
13:35:20:MultiModeLLM: - Installing the CodeProject.AI SDK...Checking ...Check done...Installing CodeProject-AI-SDK...Collecting CodeProject-AI-SDK
13:35:20:MultiModeLLM: Using cached codeproject_ai_sdk-0.0.21-py3-none-any.whl.metadata (931 bytes)
13:35:21:MultiModeLLM: Collecting Pillow (from CodeProject-AI-SDK)
13:35:21:MultiModeLLM: Downloading pillow-11.0.0-cp310-cp310-manylinux_2_28_x86_64.whl.metadata (9.1 kB)
13:35:21:MultiModeLLM: Collecting charset-normalizer (from CodeProject-AI-SDK)
13:35:21:MultiModeLLM: Using cached charset_normalizer-3.4.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (34 kB)
13:35:21:MultiModeLLM: Collecting commentjson (from CodeProject-AI-SDK)
13:35:21:MultiModeLLM: Using cached commentjson-0.9.0.tar.gz (8.7 kB)
13:35:21:MultiModeLLM: Installing build dependencies: started
13:35:23:MultiModeLLM: Installing build dependencies: finished with status 'done'
13:35:23:MultiModeLLM: Getting requirements to build wheel: started
13:35:23:MultiModeLLM: Getting requirements to build wheel: finished with status 'done'
13:35:23:MultiModeLLM: Preparing metadata (pyproject.toml): started
13:35:23:MultiModeLLM: Preparing metadata (pyproject.toml): finished with status 'done'
13:35:24:MultiModeLLM: Collecting aiohttp (from CodeProject-AI-SDK)
13:35:24:MultiModeLLM: Downloading aiohttp-3.11.10-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (7.7 kB)
13:35:25:MultiModeLLM: Collecting aiofiles (from CodeProject-AI-SDK)
13:35:25:MultiModeLLM: Using cached aiofiles-24.1.0-py3-none-any.whl.metadata (10 kB)
13:35:25:MultiModeLLM: Collecting py-cpuinfo (from CodeProject-AI-SDK)
13:35:25:MultiModeLLM: Using cached py_cpuinfo-9.0.0-py3-none-any.whl.metadata (794 bytes)
13:35:25:MultiModeLLM: Collecting requests (from CodeProject-AI-SDK)
13:35:25:MultiModeLLM: Using cached requests-2.32.3-py3-none-any.whl.metadata (4.6 kB)
13:35:25:MultiModeLLM: Collecting aiohappyeyeballs>=2.3.0 (from aiohttp->CodeProject-AI-SDK)
13:35:25:MultiModeLLM: Using cached aiohappyeyeballs-2.4.4-py3-none-any.whl.metadata (6.1 kB)
13:35:25:MultiModeLLM: Collecting aiosignal>=1.1.2 (from aiohttp->CodeProject-AI-SDK)
13:35:25:MultiModeLLM: Using cached aiosignal-1.3.1-py3-none-any.whl.metadata (4.0 kB)
13:35:25:MultiModeLLM: Collecting async-timeout<6.0,>=4.0 (from aiohttp->CodeProject-AI-SDK)
13:35:25:MultiModeLLM: Using cached async_timeout-5.0.1-py3-none-any.whl.metadata (5.1 kB)
13:35:25:MultiModeLLM: Collecting attrs>=17.3.0 (from aiohttp->CodeProject-AI-SDK)
13:35:25:MultiModeLLM: Using cached attrs-24.2.0-py3-none-any.whl.metadata (11 kB)
13:35:25:MultiModeLLM: Collecting frozenlist>=1.1.1 (from aiohttp->CodeProject-AI-SDK)
13:35:25:MultiModeLLM: Downloading frozenlist-1.5.0-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (13 kB)
13:35:26:MultiModeLLM: Collecting multidict<7.0,>=4.5 (from aiohttp->CodeProject-AI-SDK)
13:35:26:MultiModeLLM: Downloading multidict-6.1.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (5.0 kB)
13:35:26:MultiModeLLM: Collecting propcache>=0.2.0 (from aiohttp->CodeProject-AI-SDK)
13:35:26:MultiModeLLM: Downloading propcache-0.2.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (9.2 kB)
13:35:26:MultiModeLLM: Collecting yarl<2.0,>=1.17.0 (from aiohttp->CodeProject-AI-SDK)
13:35:26:MultiModeLLM: Downloading yarl-1.18.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (69 kB)
13:35:26:MultiModeLLM: Collecting lark-parser<0.8.0,>=0.7.1 (from commentjson->CodeProject-AI-SDK)
13:35:26:MultiModeLLM: Using cached lark-parser-0.7.8.tar.gz (276 kB)
13:35:26:MultiModeLLM: Installing build dependencies: started
13:35:28:MultiModeLLM: Installing build dependencies: finished with status 'done'
13:35:28:MultiModeLLM: Getting requirements to build wheel: started
13:35:29:MultiModeLLM: Getting requirements to build wheel: finished with status 'done'
13:35:29:MultiModeLLM: Preparing metadata (pyproject.toml): started
13:35:29:MultiModeLLM: Preparing metadata (pyproject.toml): finished with status 'done'
13:35:29:MultiModeLLM: Collecting idna<4,>=2.5 (from requests->CodeProject-AI-SDK)
13:35:29:MultiModeLLM: Using cached idna-3.10-py3-none-any.whl.metadata (10 kB)
13:35:29:MultiModeLLM: Collecting urllib3<3,>=1.21.1 (from requests->CodeProject-AI-SDK)
13:35:29:MultiModeLLM: Using cached urllib3-2.2.3-py3-none-any.whl.metadata (6.5 kB)
13:35:29:MultiModeLLM: Collecting certifi>=2017.4.17 (from requests->CodeProject-AI-SDK)
13:35:29:MultiModeLLM: Using cached certifi-2024.8.30-py3-none-any.whl.metadata (2.2 kB)
13:35:29:MultiModeLLM: Collecting typing-extensions>=4.1.0 (from multidict<7.0,>=4.5->aiohttp->CodeProject-AI-SDK)
13:35:29:MultiModeLLM: Using cached typing_extensions-4.12.2-py3-none-any.whl.metadata (3.0 kB)
13:35:29:MultiModeLLM: Using cached codeproject_ai_sdk-0.0.21-py3-none-any.whl (39 kB)
13:35:29:MultiModeLLM: Using cached aiofiles-24.1.0-py3-none-any.whl (15 kB)
13:35:29:MultiModeLLM: Downloading aiohttp-3.11.10-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.6 MB)
13:35:29:MultiModeLLM: ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.6/1.6 MB 25.2 MB/s eta 0:00:00
13:35:29:MultiModeLLM: Using cached charset_normalizer-3.4.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (144 kB)
13:35:29:MultiModeLLM: Downloading pillow-11.0.0-cp310-cp310-manylinux_2_28_x86_64.whl (4.4 MB)
13:35:29:MultiModeLLM: ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.4/4.4 MB 37.8 MB/s eta 0:00:00
13:35:29:MultiModeLLM: Using cached py_cpuinfo-9.0.0-py3-none-any.whl (22 kB)
13:35:29:MultiModeLLM: Using cached requests-2.32.3-py3-none-any.whl (64 kB)
13:35:29:MultiModeLLM: Using cached aiohappyeyeballs-2.4.4-py3-none-any.whl (14 kB)
13:35:29:MultiModeLLM: Using cached aiosignal-1.3.1-py3-none-any.whl (7.6 kB)
13:35:29:MultiModeLLM: Using cached async_timeout-5.0.1-py3-none-any.whl (6.2 kB)
13:35:29:MultiModeLLM: Using cached attrs-24.2.0-py3-none-any.whl (63 kB)
13:35:29:MultiModeLLM: Using cached certifi-2024.8.30-py3-none-any.whl (167 kB)
13:35:29:MultiModeLLM: Downloading frozenlist-1.5.0-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (241 kB)
13:35:29:MultiModeLLM: Using cached idna-3.10-py3-none-any.whl (70 kB)
13:35:29:MultiModeLLM: Downloading multidict-6.1.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (124 kB)
13:35:29:MultiModeLLM: Downloading propcache-0.2.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (205 kB)
13:35:29:MultiModeLLM: Using cached urllib3-2.2.3-py3-none-any.whl (126 kB)
13:35:29:MultiModeLLM: Downloading yarl-1.18.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (319 kB)
13:35:29:MultiModeLLM: Using cached typing_extensions-4.12.2-py3-none-any.whl (37 kB)
13:35:30:MultiModeLLM: Building wheels for collected packages: commentjson, lark-parser
13:35:30:MultiModeLLM: Building wheel for commentjson (pyproject.toml): started
13:35:30:MultiModeLLM: Building wheel for commentjson (pyproject.toml): finished with status 'done'
13:35:30:MultiModeLLM: Created wheel for commentjson: filename=commentjson-0.9.0-py3-none-any.whl size=12076 sha256=d9722bbb88d7692ac2ca3b5ba53d3c576a0ce18bb1f62d181692a31581b751f4
13:35:30:MultiModeLLM: Stored in directory: /root/.cache/pip/wheels/7d/90/23/6358a234ca5b4ec0866d447079b97fedf9883387d1d7d074e5
13:35:30:MultiModeLLM: Building wheel for lark-parser (pyproject.toml): started
13:35:30:MultiModeLLM: Building wheel for lark-parser (pyproject.toml): finished with status 'done'
13:35:30:MultiModeLLM: Created wheel for lark-parser: filename=lark_parser-0.7.8-py2.py3-none-any.whl size=62511 sha256=d54e98a52405061948ca7d48bf8f8fd9d45656b9a0262953b3eedab413ba751b
13:35:30:MultiModeLLM: Stored in directory: /root/.cache/pip/wheels/29/30/94/33e8b58318aa05cb1842b365843036e0280af5983abb966b83
13:35:30:MultiModeLLM: Successfully built commentjson lark-parser
13:35:30:MultiModeLLM: Installing collected packages: py-cpuinfo, lark-parser, urllib3, typing-extensions, propcache, Pillow, idna, frozenlist, commentjson, charset-normalizer, certifi, attrs, async-timeout, aiohappyeyeballs, aiofiles, requests, multidict, aiosignal, yarl, aiohttp, CodeProject-AI-SDK
13:35:32:MultiModeLLM: Successfully installed CodeProject-AI-SDK-0.0.21 Pillow-11.0.0 aiofiles-24.1.0 aiohappyeyeballs-2.4.4 aiohttp-3.11.10 aiosignal-1.3.1 async-timeout-5.0.1 attrs-24.2.0 certifi-2024.8.30 charset-normalizer-3.4.0 commentjson-0.9.0 frozenlist-1.5.0 idna-3.10 lark-parser-0.7.8 multidict-6.1.0 propcache-0.2.1 py-cpuinfo-9.0.0 requests-2.32.3 typing-extensions-4.12.2 urllib3-2.2.3 yarl-1.18.3
13:35:32:MultiModeLLM: WARNING: Target directory /app/modules/MultiModeLLM/bin/ubuntu/python310/venv/lib/python3.10/site-packages/charset_normalizer already exists. Specify --upgrade to force replacement.
13:35:32:MultiModeLLM: WARNING: Target directory /app/modules/MultiModeLLM/bin/ubuntu/python310/venv/lib/python3.10/site-packages/idna-3.10.dist-info already exists. Specify --upgrade to force replacement.
13:35:32:MultiModeLLM: WARNING: Target directory /app/modules/MultiModeLLM/bin/ubuntu/python310/venv/lib/python3.10/site-packages/certifi already exists. Specify --upgrade to force replacement.
13:35:32:MultiModeLLM: WARNING: Target directory /app/modules/MultiModeLLM/bin/ubuntu/python310/venv/lib/python3.10/site-packages/pycache already exists. Specify --upgrade to force replacement.
13:35:32:MultiModeLLM: WARNING: Target directory /app/modules/MultiModeLLM/bin/ubuntu/python310/venv/lib/python3.10/site-packages/typing_extensions-4.12.2.dist-info already exists. Specify --upgrade to force replacement.
13:35:32:MultiModeLLM: WARNING: Target directory /app/modules/MultiModeLLM/bin/ubuntu/python310/venv/lib/python3.10/site-packages/urllib3-2.2.3.dist-info already exists. Specify --upgrade to force replacement.
13:35:32:MultiModeLLM: WARNING: Target directory /app/modules/MultiModeLLM/bin/ubuntu/python310/venv/lib/python3.10/site-packages/charset_normalizer-3.4.0.dist-info already exists. Specify --upgrade to force replacement.
13:35:32:MultiModeLLM: WARNING: Target directory /app/modules/MultiModeLLM/bin/ubuntu/python310/venv/lib/python3.10/site-packages/certifi-2024.8.30.dist-info already exists. Specify --upgrade to force replacement.
13:35:32:MultiModeLLM: WARNING: Target directory /app/modules/MultiModeLLM/bin/ubuntu/python310/venv/lib/python3.10/site-packages/requests-2.32.3.dist-info already exists. Specify --upgrade to force replacement.
13:35:32:MultiModeLLM: WARNING: Target directory /app/modules/MultiModeLLM/bin/ubuntu/python310/venv/lib/python3.10/site-packages/idna already exists. Specify --upgrade to force replacement.
13:35:32:MultiModeLLM: WARNING: Target directory /app/modules/MultiModeLLM/bin/ubuntu/python310/venv/lib/python3.10/site-packages/typing_extensions.py already exists. Specify --upgrade to force replacement.
13:35:32:MultiModeLLM: WARNING: Target directory /app/modules/MultiModeLLM/bin/ubuntu/python310/venv/lib/python3.10/site-packages/requests already exists. Specify --upgrade to force replacement.
13:35:32:MultiModeLLM: WARNING: Target directory /app/modules/MultiModeLLM/bin/ubuntu/python310/venv/lib/python3.10/site-packages/urllib3 already exists. Specify --upgrade to force replacement.
13:35:32:MultiModeLLM: WARNING: Target directory /app/modules/MultiModeLLM/bin/ubuntu/python310/venv/lib/python3.10/site-packages/bin already exists. Specify --upgrade to force replacement.
13:35:33:MultiModeLLM: (✅ checked) done
13:35:33:MultiModeLLM: Scanning modulesettings for downloadable models...No models specified
13:35:33:MultiModeLLM: SELF TEST START ======================================================
13:35:33:MultiModeLLM: Info: Using PIL for image manipulation (Either OpenCV or numpy not available for this module)
13:35:33:MultiModeLLM: Traceback (most recent call last):
13:35:33:MultiModeLLM: File "/app/modules/MultiModeLLM/bin/ubuntu/python310/venv/lib/python3.10/site-packages/onnxruntime_genai/init.py", line 12, in
13:35:33:MultiModeLLM: from onnxruntime_genai.onnxruntime_genai import *
13:35:33:MultiModeLLM: ImportError: libcudnn.so.9: cannot open shared object file: No such file or directory
13:35:33:MultiModeLLM: During handling of the above exception, another exception occurred:
13:35:33:MultiModeLLM: Traceback (most recent call last):
13:35:33:MultiModeLLM: File "/app/modules/MultiModeLLM/multimode_llm_adapter.py", line 16, in
13:35:33:MultiModeLLM: from multimode_llm import MultiModeLLM, accel_mode
13:35:33:MultiModeLLM: File "/app/modules/MultiModeLLM/multimode_llm.py", line 16, in
13:35:33:MultiModeLLM: import onnxruntime_genai as og
13:35:33:MultiModeLLM: File "/app/modules/MultiModeLLM/bin/ubuntu/python310/venv/lib/python3.10/site-packages/onnxruntime_genai/init.py", line 20, in
13:35:33:MultiModeLLM: from onnxruntime_genai.onnxruntime_genai import *
13:35:33:MultiModeLLM: ImportError: libcudnn.so.9: cannot open shared object file: No such file or directory
13:35:33:MultiModeLLM: Self-test failed
13:35:33:MultiModeLLM: SELF TEST END ======================================================
13:35:33:MultiModeLLM: Module setup time 00:02:35
13:35:33:MultiModeLLM: Install failed: Self-test failed
13:35:33:MultiModeLLM: Setup complete
13:35:33:MultiModeLLM: Total setup time 00:02:39
13:35:33:MultiModeLLM: SETUP FAILED:
13:35:33:MultiModeLLM: - [Int: MultiModeLLM] Self-test failed
13:35:33:Module MultiModeLLM installed successfully.
13:35:33:Installer exited with code 1
13:35:33:Module MultiModeLLM not configured to AutoStart.

[ChrisMaunder]

Could you try something for me please: open a terminal into your running docker container (Docker Desktop or VS Code with the remote extension allows thjs) and then go to app/modules/MultiModeLLM and run bash ../../setup.sh and let me know what happens

[ac2323]

There is no setup.sh, but there is install.sh. When executing install.sh, I get the error: This script is only called from: bash ../../CodeProject.AI-Server/src/setup.sh

[ChrisMaunder]

is there a setup.sh in the /app folder? I assume there is a /app/modules/MultiModeLLM folder. If this is the case then bash ../../setup.sh should work

[ac2323]

Got the same error:

Hi Docker! We will disable shared python installs for downloaded modules


             Installing CodeProject.AI Analysis Module

======================================================================

                   CodeProject.AI Installer

======================================================================

11.06 GiB of 88.09 GiB available on Docker (linux ubuntu x86_64 - linux)
Installing xz-utils...
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
0 upgraded, 0 newly installed, 0 to remove and 47 not upgraded.


General CodeProject.AI setup

Setting permissions on runtimes folder...done
Setting permissions on general downloads folder...done
Setting permissions on module asset downloads folder...done
Setting permissions on modules folder...done
Setting permissions on models folder...done
Setting permissions on persisted data folder...done

GPU support

CUDA (NVIDIA) Present: Yes (CUDA 12.4, cuDNN 8.9.6)
ROCm (AMD) Present:    No
MPS (Apple) Present:   No

Reading module settings.......done
Processing module MultiModeLLM 1.1.0 (Internal)

Installing Python 3.10
Python 3.10 is already installed
Ensuring PIP in base python install...done
Upgrading PIP in base python install...done
Virtual Environment already present
Checking for Python 3.10...(Found Python 3.10.12) All good
Upgrading PIP in virtual environment...done
Installing updated setuptools in venv...done
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following additional packages will be installed:
  git git-man less libcbor0.8 liberror-perl libfido2-1 openssh-client xauth
Suggested packages:
  gettext-base git-daemon-run | git-daemon-sysvinit git-doc git-email git-gui gitk gitweb git-cvs git-mediawiki git-svn keychain libpam-ssh monkeysphere ssh-askpass
The following NEW packages will be installed:
  git git-lfs git-man less libcbor0.8 liberror-perl libfido2-1 openssh-client xauth
0 upgraded, 9 newly installed, 0 to remove and 47 not upgraded.
Need to get 8873 kB of archives.
After this operation, 35.4 MB of additional disk space will be used.
Do you want to continue? [Y/n] y
Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 less amd64 590-1ubuntu0.22.04.3 [142 kB]
Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 libcbor0.8 amd64 0.8.0-2ubuntu1 [24.6 kB]
Get:3 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfido2-1 amd64 1.10.0-1 [82.8 kB]
Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 openssh-client amd64 1:8.9p1-3ubuntu0.10 [906 kB]
Get:5 http://archive.ubuntu.com/ubuntu jammy/main amd64 xauth amd64 1:1.1-1build2 [27.5 kB]
Get:6 http://archive.ubuntu.com/ubuntu jammy/main amd64 liberror-perl all 0.17029-1 [26.5 kB]
Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 git-man all 1:2.34.1-1ubuntu1.11 [955 kB]
Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 git amd64 1:2.34.1-1ubuntu1.11 [3165 kB]
Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 git-lfs amd64 3.0.2-1ubuntu0.3 [3544 kB]
Fetched 8873 kB in 1s (8380 kB/s)
debconf: delaying package configuration, since apt-utils is not installed
Selecting previously unselected package less.
(Reading database ... 31209 files and directories currently installed.)
Preparing to unpack .../0-less_590-1ubuntu0.22.04.3_amd64.deb ...
Unpacking less (590-1ubuntu0.22.04.3) ...
Selecting previously unselected package libcbor0.8:amd64.
Preparing to unpack .../1-libcbor0.8_0.8.0-2ubuntu1_amd64.deb ...
Unpacking libcbor0.8:amd64 (0.8.0-2ubuntu1) ...
Selecting previously unselected package libfido2-1:amd64.
Preparing to unpack .../2-libfido2-1_1.10.0-1_amd64.deb ...
Unpacking libfido2-1:amd64 (1.10.0-1) ...
Selecting previously unselected package openssh-client.
Preparing to unpack .../3-openssh-client_1%3a8.9p1-3ubuntu0.10_amd64.deb ...
Unpacking openssh-client (1:8.9p1-3ubuntu0.10) ...
Selecting previously unselected package xauth.
Preparing to unpack .../4-xauth_1%3a1.1-1build2_amd64.deb ...
Unpacking xauth (1:1.1-1build2) ...
Selecting previously unselected package liberror-perl.
Preparing to unpack .../5-liberror-perl_0.17029-1_all.deb ...
Unpacking liberror-perl (0.17029-1) ...
Selecting previously unselected package git-man.
Preparing to unpack .../6-git-man_1%3a2.34.1-1ubuntu1.11_all.deb ...
Unpacking git-man (1:2.34.1-1ubuntu1.11) ...
Selecting previously unselected package git.
Preparing to unpack .../7-git_1%3a2.34.1-1ubuntu1.11_amd64.deb ...
Unpacking git (1:2.34.1-1ubuntu1.11) ...
Selecting previously unselected package git-lfs.
Preparing to unpack .../8-git-lfs_3.0.2-1ubuntu0.3_amd64.deb ...
Unpacking git-lfs (3.0.2-1ubuntu0.3) ...
Setting up libcbor0.8:amd64 (0.8.0-2ubuntu1) ...
Setting up less (590-1ubuntu0.22.04.3) ...
Setting up liberror-perl (0.17029-1) ...
Setting up xauth (1:1.1-1build2) ...
Setting up git-man (1:2.34.1-1ubuntu1.11) ...
Setting up libfido2-1:amd64 (1.10.0-1) ...
Setting up openssh-client (1:8.9p1-3ubuntu0.10) ...
update-alternatives: using /usr/bin/ssh to provide /usr/bin/rsh (rsh) in auto mode
update-alternatives: warning: skip creation of /usr/share/man/man1/rsh.1.gz because associated file /usr/share/man/man1/ssh.1.gz (of link group rsh) doesn't exist
update-alternatives: using /usr/bin/slogin to provide /usr/bin/rlogin (rlogin) in auto mode
update-alternatives: warning: skip creation of /usr/share/man/man1/rlogin.1.gz because associated file /usr/share/man/man1/slogin.1.gz (of link group rlogin) doesn't exist
update-alternatives: using /usr/bin/scp to provide /usr/bin/rcp (rcp) in auto mode
update-alternatives: warning: skip creation of /usr/share/man/man1/rcp.1.gz because associated file /usr/share/man/man1/scp.1.gz (of link group rcp) doesn't exist
Setting up git (1:2.34.1-1ubuntu1.11) ...
Setting up git-lfs (3.0.2-1ubuntu0.3) ...
Processing triggers for libc-bin (2.35-0ubuntu3.4) ...
Processing triggers for mailcap (3.70+nmu1ubuntu1) ...
Looking for model: microsoft/Phi-3-vision-128k-instruct-onnx-cuda in cuda-int4-rtn-block-32... already downloaded.
Installing Python packages for MultiModeLLM
Installing GPU-enabled libraries: If available
Searching for python3-pip...All good.
Ensuring PIP compatibility...done
Python packages will be specified by requirements.cuda12.txt
  - Installing NumPy, a package for scientific computing...Already installed
  - Installing onnxruntime-genai for CUDA, the ONNX Runtime generate() API...Already installed
  - Installing the CodeProject.AI SDK...Already installed
Scanning modulesettings for downloadable models...No models specified
Self test: Traceback (most recent call last):
  File "/app/modules/MultiModeLLM/bin/ubuntu/python310/venv/lib/python3.10/site-packages/onnxruntime_genai/__init__.py", line 12, in <module>
    from onnxruntime_genai.onnxruntime_genai import *
ImportError: libcudnn.so.9: cannot open shared object file: No such file or directory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/app/modules/MultiModeLLM/multimode_llm_adapter.py", line 16, in <module>
    from multimode_llm import MultiModeLLM, accel_mode
  File "/app/modules/MultiModeLLM/multimode_llm.py", line 16, in <module>
    import onnxruntime_genai as og
  File "/app/modules/MultiModeLLM/bin/ubuntu/python310/venv/lib/python3.10/site-packages/onnxruntime_genai/__init__.py", line 20, in <module>
    from onnxruntime_genai.onnxruntime_genai import *
ImportError: libcudnn.so.9: cannot open shared object file: No such file or directory
Self-test failed
Module setup time 00:00:28
Install failed: Self-test failed

                Setup complete

Total setup time 00:00:32

SETUP FAILED:

 - [Int: MultiModeLLM] Self-test failed

[ChrisMaunder]

ImportError: libcudnn.so.9: cannot open shared object file: No such file or directory

It looks like it's looking for cuDNN 9.X even though the official docs say 9.X isn't supported. I'm stumped. It installed fine on Linux, CUDA 12, cuDNN 8 natively. I'm not sure what to suggest here other than maybe try the CUDA 11.8 Docker image as a wild stab in the dark on this one.