LiteLLM Proxy + MCP Integration

This setup provides a unified OpenAI-compatible API gateway with working MCP (Model Context Protocol) tools integration through the litellm-proxy and mcp-proxy proxies.

Architecture

┌─────────────────┐    ┌──────────────────┐    ┌────────────────┐
│   Client Apps   │────│  LiteLLM Proxy   │────│     vLLM       │
│                 │    │  (Port 4000)     │    │   OpenRouter   │
│                 │    └──────────────────┘    └────────────────┘
│                 │              │
│                 │              │ (Integration Planned)
│                 │              ▼
│                 │    ┌──────────────────┐    ┌────────────────┐
│                 │────│   MCP Proxy      │────│ Tavily Search  │
└─────────────────┘    │  (Port 4004)     │    │ + Other Tools  │
                       └──────────────────┘    └────────────────┘

🚀 Quick Start

1. Environment Setup

Copy the example environment file and configure your API keys:

cp .env.example .env

Edit .env and set:

• LITELLM_MASTER_KEY: Your proxy admin key (start with sk-)
• OPENROUTER_API_KEY: Your OpenRouter API key
• TAVILY_API_KEY: Your Tavily API key for web search MCP tools

2. Start All Services

# Start both LiteLLM and MCP Proxy services
docker-compose up -d

This will start:

• LiteLLM Proxy on port 4000 (OpenAI-compatible API)
• MCP Proxy on port 4004 (Working MCP tools)

3. Test the Setup

# Test LiteLLM models
chmod +x test_models.sh
./test_models.sh

# Test MCP tools directly (working approach)
chmod +x test_mcp_proxy.sh
./test_mcp_proxy.sh

🔧 Available Services

LLM Models (LiteLLM Proxy - Port 4000)

Model Name	Type	Endpoint	Status
vllm-Qwen2.5-VL-32B	Internal	10.0.0.10:7113	✅ Working
openrouter-qwen3-32b	External	OpenRouter	✅ Working

MCP Tools (MCP Proxy - Port 4004)

Tool Name	Description	Status
tavily-search	Web search via Tavily API	✅ Working
tavily-extract	Extract content from URLs	✅ Working
tavily-crawl	Crawl websites for data	✅ Working
tavily-map	Map and structure information	✅ Working

✅ Working Endpoint: http://localhost:4004/tavily/mcp

📋 Current Status

✅ What's Working

• LiteLLM Proxy: Full OpenAI-compatible API for LLM models
• MCP Proxy: Direct access to Tavily search tools via HTTP JSON-RPC
• Independent Operation: Both services work independently

⚠️ Known Issues

• LiteLLM MCP Integration (BETA): Currently has connectivity issues
• Auto-discovery: LiteLLM doesn't auto-discover MCP tools from mcp-proxy yet

🔄 Workaround

Use MCP tools directly through the mcp-proxy endpoints (see examples below).

🛠️ Usage Examples

1. LLM Chat Completions (Working)

curl -X POST http://localhost:4000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -d '{
    "model": "vllm-Qwen2.5-VL-32B",
    "messages": [
      {"role": "user", "content": "Hello! How are you?"}
    ]
  }'

2. MCP Tools (Working - Direct Access)

# Search for current events using Tavily
curl -X POST http://localhost:4004/tavily/mcp \
  -H "Content-Type: application/json" \
  -d '{
    "jsonrpc": "2.0",
    "id": 1,
    "method": "tools/call",
    "params": {
      "name": "tavily-search",
      "arguments": {
        "query": "latest AI news 2025"
      }
    }
  }'

3. List Available MCP Tools

# List all available MCP tools
curl -X POST http://localhost:4004/tavily/mcp \
  -H "Content-Type: application/json" \
  -d '{
    "jsonrpc": "2.0",
    "id": 1,
    "method": "tools/list"
  }'

🔗 Integration Options

Option 1: Direct MCP Access (Recommended)

Use the mcp-proxy directly for reliable tool access:

• Endpoint: http://localhost:4004/tavily/mcp
• Transport: HTTP JSON-RPC
• Auth: None required

Option 2: Combined Workflow

1. Use LiteLLM for LLM inference
2. Use MCP Proxy for tool calls
3. Combine results in your application logic

Option 3: Future Integration

Once LiteLLM's MCP beta issues are resolved, full integration will be available.

📚 Documentation

• MCP.md - Detailed MCP Proxy setup and configuration
• LiteLLM Docs - Official LiteLLM documentation

📁 File Structure

├── docker-compose.yml          # Both LiteLLM and MCP Proxy services
├── config.yaml                 # LiteLLM configuration  
├── mcp-proxy-config.json       # MCP Proxy configuration
├── .env                        # Environment variables
├── test_models.sh              # LiteLLM test script
├── test_mcp_proxy.sh           # MCP Proxy test script
├── test_mcp_proxy2.sh          # Additional MCP tests
├── README.md                   # This file
└── MCP.md                      # Detailed MCP setup guide

For detailed MCP Proxy setup and configuration, see MCP.md.