privateGPT installation guide on Debian 11

[raymond0517]

is there any link providing detail step by step guide to install privateGPT on debian 11?

Thank you

Raymond

[000Shimmy]

Debian 13 (testing) Install Notes

Here's a verbose copy of my install notes using the latest version of Debian 13 (Testing) a.k.a Trixie and the 6.5.x kernel. Also, keep in mind that PrivateGPT is cutting edge FOSS tech and it may not play nice with older systems; particularly older GPUs. That being said your mileage may vary depending on your hardware; so, per usual nvidia GPUs make you fight for the FOSS life.

Intro

By default, Trixie doesn't trust snakes; she requires that anything Python be installed via apt install python3-{name}. However, some - read "most" - of the python3 tools/apps/utils/etc are not available in the Debian repos. If you want to use any of those questionable snakes then they must be used within a pre-built virtual environment. There exists great arguments for and against this approach: I'll leave you to your opinions, and get on with the Debian way of installing PrivateGPT. When done you should have a PrivateGPT instance up and running on your machine. Let's get to it then; shall we?

This is the Debian Way

sudo apt update 
sudo apt upgrade 

Install PrivateGPT

Change to the directory that you want to install the virtual python environment for PrivateGPT into. In this example I will be using the Desktop directory, but you can use anyone that you like.

cd ~/Desktop 
git clone "https://github.com/imartinez/privateGPT.git"

To accommodate the Debian virtual environment requisite we have to deviate from the standard instructions just a bit. Don't change into the privateGPT directory just yet. First, as required by picky Trixie, you have to build and activate the virtual environment. Make sure that if you use GPG Wallet, such as KDE's kwallet; that you are signed into it or all the poetry dependencies install will fail.

python3 -m venv ./privateGPT       # build the virtual environment 
source ./privateGPT/bin/activate   # activate the virtual environment 

You'll notice a change in your PS1 prompt. After you cd into the privateGPT directory you will be inside the virtual environment that you just built and activated for it. Use the deactivate command to shut it down. NOTE: with entr or another tool you can automate most activating and deactivating the virtual environment, along with starting the privateGPT server with a couple of scripts; even dmenu offers some viable actions, but that's off-topic though.

Some of the dependencies and language model files installed by poetry are quite large and depending upon your ISP's bandwidth speeds this part may take awhile. If you haven't had your coffee, or tea, warmed up in a while then immediately following this next set up commands will be the time to do accomplish that task.

cd ./privateGPT  
pip install poetry                           # installs the version control installer 
poetry install --with ui                     # install dependencies
poetry run python scripts/setup              # installs models

When that's done you will have access to your own privateGPT available at localhost:8001 or 127.0.0.1:8001 after you run the following command.

poetry run python -m private_gpt   # runs the privateGPT server

Back-Grounding PrivateGPT

After you get privateGPT up and running, test it out with some documents. Then you can run it in the background with the following command. NOTE: stdout redirects to /dev/null so all your goodness in your output is sent into the unknown abyss.

poetry run python -m private_gpt > /dev/null 2>&1 & # logs into the abyss & and regain use of the terminal 

I hope this helped and may God bless you.

[pierri1101]

poetry install --with ui didn't work for me.

Instead I used: poetry install --extras "ui"

[janwarming]

Will it support more than one GPU, I have a mining rig I like to use for AI.

[000Shimmy]

The python environment encapsulates the python operations of the privateGPT within the directory, but it’s not a container in the sense of podman or lxc. I don’t foresee any “breaking” issues assigning privateGPT more than one GPU from the OS as described in the docs. However, I don’t have any surplus GPUs at the moment to test this theory out, so…

If you can’t get it to work, post back and we will figure something else out. There’s more than one way to accomplish your goals.

[reefreaper]

Thank you for this how to. I'm up and running with Kali latest and it works great. A little slow but I'm running on CPU.