Nexa SDK is an on-device inference framework that runs any model on any device, across any backend. It runs on CPUs and GPUs with backend support for CUDA, Metal, and Vulkan. It handles multiple input modalities including text 📝, image 🖼️, and audio 🎧. The SDK includes an OpenAI-compatible API server with support for JSON schema-based function calling and streaming. It supports model formats such as GGUF and MLX, enabling efficient quantized inference across diverse platforms.
- MLX and GGUF support
- VLM and LLMs
curl -fsSL https://raw.githubusercontent.com/NexaAI/nexa-sdk/main/release/linux/install.sh -o install.sh && chmod +x install.sh && ./install.shYou can run any compatible GGUF or MLX model from 🤗 Hugging Face by using the <full repo name>.
Tip
GGUF runs on macOS, Linux, and Windows.
📝 Run and chat with LLMs, e.g. Qwen3:
nexa infer ggml-org/Qwen3-1.7B-GGUF🖼️ Run and chat with Multimodal models, e.g. Qwen2.5-Omni:
nexa infer NexaAI/Qwen2.5-Omni-3B-GGUFTip
MLX is macOS-only (Apple Silicon). Many MLX models in the Hugging Face mlx-community organization have quality issues and may not run reliably. We recommend starting with models from our curated NexaAI Collection for best results. For example
📝 Run and chat with LLMs, e.g. Qwen3:
nexa infer NexaAI/Qwen3-4B-4bit-MLX🖼️ Run and chat with Multimodal models, e.g. Gemma3n:
nexa infer NexaAI/gemma-3n-E4B-it-4bit-MLX| Essential Command | What it does |
|---|---|
nexa -h |
show all CLI commands |
nexa pull <repo> |
Interactive download & cache of a model |
nexa infer <repo> |
Local inference |
nexa list |
Show all cached models with sizes |
nexa remove <repo> / nexa clean |
Delete one / all cached models |
nexa serve --host 127.0.0.1:8080 |
Launch OpenAI‑compatible REST server |
nexa run <repo> |
Chat with a model via an existing server |
👉 To interact with multimodal models, you can drag photos or audio clips directly into the CLI — you can even drop multiple images at once!
See CLI Reference for full commands.
We would like to thank the following projects: