Skip to content

Commit 04eb718

Browse files
committed
update docs
1 parent 7bf6769 commit 04eb718

File tree

2 files changed

+56
-15
lines changed

2 files changed

+56
-15
lines changed

docs/my-website/docs/proxy/alerting.md

Lines changed: 56 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -6,17 +6,13 @@ import TabItem from '@theme/TabItem';
66

77
Get alerts for:
88

9-
- Hanging LLM api calls
10-
- Slow LLM api calls
11-
- Failed LLM api calls
12-
- Budget Tracking per key/user
13-
- Spend Reports - Weekly & Monthly spend per Team, Tag
14-
- Failed db read/writes
15-
- Model outage alerting
16-
- Daily Reports:
17-
- **LLM** Top 5 slowest deployments
18-
- **LLM** Top 5 deployments with most failed requests
19-
- **Spend** Weekly & Monthly spend per Team, Tag
9+
| Category | Alert Type |
10+
|----------|------------|
11+
| **LLM Performance** | Hanging API calls, Slow API calls, Failed API calls, Model outage alerting |
12+
| **Budget & Spend** | Budget tracking per key/user, Soft budget alerts, Weekly & Monthly spend reports per Team/Tag |
13+
| **System Health** | Failed database read/writes |
14+
| **Daily Reports** | Top 5 slowest LLM deployments, Top 5 LLM deployments with most failed requests, Weekly & Monthly spend per Team/Tag |
15+
2016

2117

2218
Works across:
@@ -93,6 +89,51 @@ litellm_settings:
9389
redact_messages_in_exceptions: True
9490
```
9591

92+
### Soft Budget Alerts for Virtual Keys
93+
94+
Use this to send an alert when a key/team is close to it's budget running out
95+
96+
Step 1. Create a virtual key with a soft budget
97+
98+
Set the `soft_budget` to 0.001
99+
100+
```shell
101+
curl -X 'POST' \
102+
'http://localhost:4000/key/generate' \
103+
-H 'accept: application/json' \
104+
-H 'x-goog-api-key: sk-1234' \
105+
-H 'Content-Type: application/json' \
106+
-d '{
107+
"key_alias": "prod-app1",
108+
"team_id": "113c1a22-e347-4506-bfb2-b320230ea414",
109+
"soft_budget": 0.001
110+
}'
111+
```
112+
113+
Step 2. Send a request to the proxy with the virtual key
114+
115+
```shell
116+
curl http://0.0.0.0:4000/chat/completions \
117+
-H "Content-Type: application/json" \
118+
-H "Authorization: Bearer sk-Nb5eCf427iewOlbxXIH4Ow" \
119+
-d '{
120+
"model": "openai/gpt-4",
121+
"messages": [
122+
{
123+
"role": "user",
124+
"content": "this is a test request, write a short poem"
125+
}
126+
]
127+
}'
128+
129+
```
130+
131+
Step 3. Check slack for Expected Alert
132+
133+
<Image img={require('../../img/soft_budget_alert.png')}/>
134+
135+
136+
96137

97138
### Add Metadata to alerts
98139

@@ -123,7 +164,7 @@ response = client.chat.completions.create(
123164

124165
<Image img={require('../../img/alerting_metadata.png')}/>
125166

126-
### Opting into specific alert types
167+
### Select specific alert types
127168

128169
Set `alert_types` if you want to Opt into only specific alert types. When alert_types is not set, all Default Alert Types are enabled.
129170

@@ -145,7 +186,7 @@ general_settings:
145186
]
146187
```
147188

148-
### Set specific slack channels per alert type
189+
### Map slack channels to alert type
149190

150191
Use this if you want to set specific channels per alert type
151192

@@ -243,7 +284,7 @@ curl -i http://localhost:4000/v1/chat/completions \
243284
```
244285

245286

246-
### Using MS Teams Webhooks
287+
### MS Teams Webhooks
247288

248289
MS Teams provides a slack compatible webhook url that you can use for alerting
249290

@@ -285,7 +326,7 @@ curl --location 'http://0.0.0.0:4000/health/services?service=slack' \
285326

286327
<Image img={require('../../img/ms_teams_alerting.png')}/>
287328

288-
### Using Discord Webhooks
329+
### Discord Webhooks
289330

290331
Discord provides a slack compatible webhook url that you can use for alerting
291332

46.9 KB
Loading

0 commit comments

Comments
 (0)