LiteLLM Service

Overview

LiteLLM Service is a shell-based toolkit for managing, testing, and running custom Ollama models and the LiteLLM proxy. It provides an interactive menu for model lifecycle management, proxy control, and integration with the Cursor editor.

Features

• Create, select, and test custom Ollama models
• Install, upgrade, and run the LiteLLM proxy
• Background proxy management with logging
• Integration with Cursor editor
• Interactive menu for all major operations

Quick Start

1. Install prerequisites:
   • Python 3.12+
   • pip
   • Conda
2. Clone the repository:

   git clone <repo-url>
   cd litellm

3. Create and activate the Conda environment:

   conda create -n litellm python=3.12
   conda activate litellm

4. Install dependencies:

   pip install -r requirements.txt
   pip install "litellm[proxy]"

5. Configure environment variables:
   • Copy src/config/.env.example to src/config/.env and update as needed.
6. Run the service:

   bash launch.sh

Interactive Workflow

When you run bash launch.sh, you'll see a menu with options:

1. Select Custom Model: Choose an Ollama model to use.
2. Create Custom Model: Build a new model from a Modelfile.
3. Test Custom Model: Send a test prompt to your model.
4. Run Custom Model: Run the selected model with Ollama.
5. Install LiteLLM Proxy: Install the proxy via pip.
6. Upgrade LiteLLM Proxy: Upgrade to the latest proxy version.
7. Start LiteLLM Proxy: Run the proxy in the foreground.
8. Start LiteLLM Proxy (BG): Run the proxy in the background with logging.
9. Test LiteLLM Proxy: Send a test request to the proxy.
10. Start Cursor: Launch the Cursor editor (if installed).
11. Stop LiteLLM Proxy: Stop the running proxy.
12. LiteLLM Proxy Status: Check if the proxy is running.
13. Exit: Quit the service.

Troubleshooting

• If you see errors about missing commands, ensure all prerequisites are installed and your environment is activated.
• For environment issues, check your .env file in src/config/.
• For proxy issues, check logs in the ./.log directory.

FAQ

Q: How do I add a new model? A: Use menu option [2] and provide the model name and Modelfile path.

Q: How do I change the proxy port? A: Edit LITELLM_PORT in src/config/.env or src/common.sh and restart the proxy.

Q: How do I integrate with Cursor? A: Use menu option [10]. Ensure Cursor is installed at the default path.

Authors

• J.K. Thysse

Resources

• Ollama Documentation
• LiteLLM Documentation