Skip to content

Commit

Permalink
Oobabooga: document the changes
Browse files Browse the repository at this point in the history
  • Loading branch information
enricoros committed Dec 8, 2023
1 parent 0cf8f04 commit f378733
Show file tree
Hide file tree
Showing 2 changed files with 21 additions and 10 deletions.
31 changes: 21 additions & 10 deletions docs/config-local-oobabooga.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ Integrate local Large Language Models (LLMs) with
[oobabooga/text-generation-webui](https://github.com/oobabooga/text-generation-webui),
a specialized interface that includes a custom variant of the OpenAI API for a smooth integration process.

_Last updated on Nov 7, 2023_
_Last updated on Dec 7, 2023_

### Components

Expand All @@ -20,26 +20,35 @@ This guide assumes that **big-AGI** is already installed on your system. Note th

### Text-web-ui Installation & Configuration:

1. Install [text-generation-webui](https://github.com/oobabooga/text-generation-webui#Installation).
- Download the one-click installer, extract it, and double-click on "start" - ~10 minutes
- Close it afterwards as we need to modify the startup flags
1. Install [text-generation-webui](https://github.com/oobabooga/text-generation-webui#Installation):
- Follow the instructions in the official page (basicall clone the repo and run a script) [~10 minutes]
- Stop the Web UI as we need to modify the startup flags to enable the OpenAI API
2. Enable the **openai extension**
- Edit `CMD_FLAGS.txt`
- Make sure that `--listen --extensions openai` is present and uncommented
- Make sure that `--listen --api` is present and uncommented
3. Restart text-generation-webui
- Double-click on "start"
- You should see something like:
```
2023-11-07 21:24:26 INFO:Loading the extension "openai"...
2023-11-07 21:24:27 INFO:OpenAI compatible API URL:
2023-12-07 21:51:21 INFO:Loading the extension "openai"...
2023-12-07 21:51:21 INFO:OpenAI-compatible API URL:
http://0.0.0.0:5000/v1
http://0.0.0.0:5000
2023-12-07 21:51:21 INFO:Loading the extension "gallery"...
INFO: Started server process [24756]
INFO: Waiting for application startup.
INFO: Application startup complete.
INFO: Uvicorn running on http://0.0.0.0:5000 (Press CTRL+C to quit)
Running on local URL: http://0.0.0.0:7860
```
- The OpenAI API is now running on port 5000, on both localhost (127.0.0.1) and your network IP address
- This shows that:
- The Web UI is running on port 7860: http://127.0.0.1:7860
- **The OpenAI API is running on port 5000: http://127.0.0.1:5000**
4. Load your first model
- Open the text-generation-webui at [127.0.0.1:7860](http://127.0.0.1:7860/)
- Switch to the **Model** tab
- Download, for instance, `TheBloke/Llama-2-7b-Chat-GPTQ:gptq-4bit-32g-actorder_True` - 4.3 GB
- Download, for instance, `TheBloke/Llama-2-7B-Chat-GPTQ`
- Select the model once it's loaded
### Integrating text-web-ui with big-AGI:
Expand All @@ -51,4 +60,6 @@ This guide assumes that **big-AGI** is already installed on your system. Note th
- The active model must be selected and LOADED on the text-generation-webui as it doesn't support model switching or parallel requests.
- Select model & Chat
![config-oobabooga-0.png](pixels/config-oobabooga-0.png)
Enjoy the privacy and flexibility of local LLMs with `big-AGI` and `text-generation-webui`!
Binary file added docs/pixels/config-oobabooga-0.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.

0 comments on commit f378733

Please sign in to comment.