Skip to content

Commit

Permalink
🔨[DEV] Refine README
Browse files Browse the repository at this point in the history
  • Loading branch information
fairyshine committed Sep 30, 2024
1 parent 270091c commit ab0413e
Showing 1 changed file with 8 additions and 6 deletions.
14 changes: 8 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,6 @@

An easy-to-use, high-performance(?) backend for serving LLMs and other AI models, built on FastAPI.

[TOC]

## ✨ 1 Features

### 1.1 Model: Support models with various backends
Expand All @@ -17,8 +15,12 @@ An easy-to-use, high-performance(?) backend for serving LLMs and other AI models
-[llama.cpp](https://github.com/abetlen/llama-cpp-python)
- `LlamacppLM` (`Llama`)

-[OpenAI](https://platform.openai.com/docs/api-reference/introduction)
- `OpenAIChatModel` (`/chat/completions`)

- [MLC LLM](https://llm.mlc.ai)
- [vllm](https://github.com/vllm-project/vllm)

- ...

### 1.2 Modules: More than just chatting with models
Expand All @@ -41,9 +43,9 @@ An easy-to-use, high-performance(?) backend for serving LLMs and other AI models
pip install fastmindapi
```

### 2.2 Usage
### 2.2 Usage (*C/S*)

#### 2.2.1 Run the server
#### 2.2.1 Run the server (*S*)

##### in Terminal

Expand All @@ -60,7 +62,7 @@ server = FM.Server(API_KEY="sk-19992001")
server.run()
```

#### 2.2.2 Access the service
#### 2.2.2 Access the service (*C*)

##### via client

Expand All @@ -80,7 +82,7 @@ curl http://127.0.0.1:8000/model/add_info \
}'

# 2. load model
curl http://127.0.0.1:8000/model/load/gemma2 -H "Authorization: Bearer sk-anything"
curl http://127.0.0.1:8000/model/load/gemma2 -H "Authorization: Bearer sk-19992001"

# 3. run model inference
# 3.1 Generation API
Expand Down

0 comments on commit ab0413e

Please sign in to comment.