Extra Things I did to get set up.

[Phoenixxl]

Hey all you lovely people, I'm writing up what I did extra to get things up and running.
I don't own a machine which runs a hypervisor with an adequate video card in it. Nor a linux desktop machine with an adequate video card.
What I do own though is my desktop with a 4090 and a second "family" computer that's in the living room which has a 2080TI. Both of these run windows.

I do have one of my servers which runs a hypervisor. I used a VM to install a minimal version of ubuntu server. This is what I usually install anyway. I installed Ollama on the living room machine. Installed an extra 980TI I had lying around as well.
On the windows machines I added a few environment variables: ( start -> type env , you will see "edit the system environment variables")
There I stopped ollama and added 2 new >system< variables.

OLLAMA_MODELS         Y:\OLLAMA\models
OLLAMA_HOST              0.0.0.0

Y being a network share. Sharing the language models across computers.
0.0.0.0 being the ip address the tcp port should bind on being "all interfaces"
When starting Ollama again I got the standard windows requester asking if it's ok to open the port to the local network , replied yes.

On the linux machine added 2 extra users beyond the one created during install. I didn't choose to use virtualenv.
On all 3 users, one by one I installed ShellGPT, it complained about the PATH var not including a bit including the per-user bin. This is due to the fact the shell startup batch .bashrc only adds these to path when they actually exist. Before the install of ShellGPT they didn't exist, so you can safely ignore the message. But, you need to log out and log back in then echo $PATH to see if they're there when you start sgpt for the first time. When you do edit ~/.config/shell_gpt/.sgptrc change the paths for cache and chat/cache to be somewhere else for each user. /tmp/username/shell_gpt/cache and /tmp/username/shell_gpt/chat_cache for example.
Me , I just chose a storage location instead of using /tmp but preload it in cache using :

vmtouch -l /home/USERNAME/.config/shell_gpt/cache/ /home/USERNAME/.config/shell_gpt/chat_cache/

Why 3 users? I made one user to use cheap ass gpt-3.5 , one user to use the latest and greatest GPT-4 , the third user to use Ollama.
For the Ollama user II followed the instructions linked on this main github page. I considered using LXD containers for the 3 users or virtualenv but this worked too, less is more.

I did modify a few things.

In files of litellm for user 3
~/.local/lib/python3.10/site-packages/litellm/llms/ollama.py
~/.local/lib/python3.10/site-packages/litellm/llms/ollama_chat.py
I did a quick hack to add support for the new Keep_alive option in Ollama so my model doesn't unload every 5 minutes. It takes ages to load a 34B on that hardware.

In those files, around line 150 I changed:

    data = {
        "model": model,
        "prompt": prompt,
        "options": optional_params,
        "stream": stream,
    }

to

    data = {
        "model": model,
        "prompt": prompt,
        "options": optional_params,
        "stream": stream,
        "keep_alive": "-1m",
    }

-1m means "don't unload" , do this for all users that use ollama.

To make Litellm connect to the windows machine running Ollama I did what I knew worked. I saw something about proxies in the litellm install but didn't really bother with it. I just used socat to redirect the local Olama port to the remote computer. It did the trick apparently.

socat tcp-listen:11434,reuseaddr,fork tcp:192.168.0.1:11434

I just put 192.168.0.1 here as an example but that's where the IP of the computer that has ollama goes.

What I do to run these batches? I make the appropriate maps in /root then install cron and edit root's crontab.
crontab -e

* * * * * /usr/bin/flock -w 0 /root/MyBatches/OllamaTunnel/OllamaTunnel.lock /root/MyBatches/OllamaTunnel/StartTunnel.sh >/dev/null 2>&1
* * * * * /usr/bin/flock -w 0 /root/MyBatches/CacheDirs/CacheDirs.lock /root/MyBatches/CacheDirs/StartVmtouch.sh >/dev/null 2>&1

What I also did was manually add this patch to make --repl and executing functions work better together.
Link

Changing the default gpt model for user 1 and user 2 is also done in
~/.config/shell_gpt/.sgptrc

Since I mainly use windows I used 3 profiles with different colored default backgrounds in Putty so I instantly know which of the 3 models I'm using.

During all this time I had a failing drive in a zpool, it was fun to feign ignorance , pipe the stdout of some commands to sgpt and eventually get explained what to do. It was even correct. go figure.

All in all it was a fairly fun install, this is a great tool and I'm sure I will enjoy adding a few more roles and try and get my ollama model to jump through more hoops and do automated tasks.

Hope this helps some other people who encounter some of these issues.