crabllm

High-performance LLM API gateway in Rust, used by OpenWalrus.

What It Does

Crabllm sits between your application and LLM providers. It exposes an OpenAI-compatible API and routes requests to the configured provider — OpenAI, Anthropic, Azure, Ollama, and any OpenAI-compatible service.

One API format. Many providers. Low overhead.

Inspired by LiteLLM. Built in Rust for minimal overhead. See the docs for providers, routing, extensions, and configuration.

Quick Start

cargo install crabllm

Create crabllm.toml:

listen = "0.0.0.0:8080"

[providers.openai]
kind = "openai_compat"
api_key = "${OPENAI_API_KEY}"
models = ["gpt-4o"]

[providers.anthropic]
kind = "anthropic"
api_key = "${ANTHROPIC_API_KEY}"
models = ["claude-sonnet-4-20250514"]

Run:

crabllm --config crabllm.toml

Send requests using the OpenAI format:

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4o",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

Why Rust

• Performance: Sub-millisecond proxy overhead. No GC pauses.
• Safety: Memory safety without runtime cost.
• Concurrency: Tokio async runtime handles thousands of concurrent streaming connections.
• Deployment: Single static binary. No interpreter, no virtualenv.

Crates

Crate	Description
crabllm	Binary entry point (CLI + server startup)
crabllm-core	Shared types: config, OpenAI-format request/response, errors
crabllm-provider	Provider enum, registry, and upstream HTTP dispatch
crabllm-proxy	Axum HTTP server, route handlers, auth middleware

License

MIT OR Apache-2.0