Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
138 changes: 138 additions & 0 deletions docs/community/engagement-plan.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,138 @@
# LangChain Community Engagement Plan

Goal: Establish AgentGuard as the go-to tool for AI agent cost control by engaging where developers already ask for help. 2-3 posts per week for the first month.

## Channels to Monitor

| Channel | URL | Frequency |
|---------|-----|-----------|
| LangChain GitHub Issues | github.com/langchain-ai/langchain/issues | Daily |
| LangChain GitHub Discussions | github.com/langchain-ai/langchain/discussions | Daily |
| LangGraph GitHub Issues | github.com/langchain-ai/langgraph/issues | 3x/week |
| r/LangChain | reddit.com/r/LangChain | 3x/week |
| r/LocalLLaMA | reddit.com/r/LocalLLaMA | 2x/week |
| LangChain Discord #general | discord.gg/langchain | Daily |
| LangChain Discord #help | discord.gg/langchain | Daily |
| CrewAI GitHub Discussions | github.com/joaomdmoura/crewAI/discussions | 2x/week |
| Hacker News (AI threads) | news.ycombinator.com | 2x/week |
| Stack Overflow [langchain] tag | stackoverflow.com/questions/tagged/langchain | 2x/week |

## 10 Pre-Researched GitHub Issues to Engage

These are recurring problem categories. Search for new instances weekly.

### 1. "Agent stuck in infinite loop"
- **Search:** `is:issue is:open label:bug "infinite loop" OR "stuck" OR "repeating"`
- **Repos:** langchain, langgraph, crewai
- **Response angle:** LoopGuard detects repeated tool calls and raises LoopDetected

### 2. "Unexpected high token usage / cost"
- **Search:** `is:issue "token usage" OR "cost" OR "expensive" OR "budget"`
- **Repos:** langchain, langgraph
- **Response angle:** BudgetGuard enforces hard dollar limits at runtime

### 3. "Agent makes too many API calls"
- **Search:** `is:issue "too many calls" OR "rate limit" OR "429" OR "max iterations"`
- **Repos:** langchain, langgraph
- **Response angle:** RateLimitGuard + BudgetGuard(max_calls=N)

### 4. "How to track costs per agent run"
- **Search:** `is:issue OR is:discussion "track cost" OR "cost per run" OR "cost tracking"`
- **Repos:** langchain, crewai
- **Response angle:** Tracer + patch_openai auto-tracks cost per call, estimate_cost() for estimates

### 5. "Agent timeout / takes too long"
- **Search:** `is:issue "timeout" OR "takes too long" OR "hanging" OR "never finishes"`
- **Repos:** langchain, langgraph, crewai
- **Response angle:** TimeoutGuard enforces wall-clock limits

### 6. "How to add callbacks / custom logging"
- **Search:** `is:issue OR is:discussion "callback" OR "custom logging" OR "trace"`
- **Repos:** langchain
- **Response angle:** AgentGuardCallbackHandler plugs into existing callback system

### 7. "LangGraph node cost tracking"
- **Search:** `is:issue "cost" OR "budget" OR "tracking"` in langgraph
- **Response angle:** guarded_node decorator wraps any node with budget + tracing

### 8. "CrewAI agent cost management"
- **Search:** `is:issue OR is:discussion "cost" OR "budget" OR "expensive"` in crewai
- **Response angle:** AgentGuardCrewHandler + step_callback integration

### 9. "CI testing for agent behavior"
- **Search:** `is:issue "ci" OR "testing" OR "evaluation" OR "regression"`
- **Repos:** langchain, langgraph
- **Response angle:** EvalSuite + CI cost gates GitHub Action

### 10. "Observability / monitoring for agents"
- **Search:** `is:issue OR is:discussion "observability" OR "monitoring" OR "dashboard"`
- **Repos:** langchain, crewai
- **Response angle:** Full tracing with zero dependencies, optional hosted dashboard

## 5 Template Responses

### Template 1: Cost overrun / budget question

> I ran into the same problem. I built [AgentGuard](https://github.com/bmdhodl/agent47) specifically for this — it lets you set a hard dollar limit on agent runs. When the budget is hit, it raises an exception and stops the agent immediately.
>
> Quick example: `BudgetGuard(max_cost_usd=5.00)` — that's it. Works with LangChain, LangGraph, CrewAI, or raw OpenAI/Anthropic. Zero dependencies.
>
> Happy to help if you have questions about integrating it.

### Template 2: Infinite loop / stuck agent

> This is a common pattern — the agent calls the same tool with the same args because it can't interpret the result. I've been working on [AgentGuard](https://github.com/bmdhodl/agent47) which has a `LoopGuard` that detects exactly this. It watches a sliding window of tool calls and raises `LoopDetected` when it sees repeats.
>
> There's also `FuzzyLoopGuard` for when the args change slightly but it's still effectively looping.

### Template 3: Cost tracking / observability

> For cost tracking, I've been using [AgentGuard](https://github.com/bmdhodl/agent47). It has built-in pricing for OpenAI, Anthropic, Google, Mistral models and auto-tracks cost when you patch the SDK client. Output goes to JSONL files or the hosted dashboard.
>
> The LangChain integration is a callback handler: `AgentGuardCallbackHandler(budget_guard=BudgetGuard(max_cost_usd=5.00))` — auto-extracts token usage from LLM responses.

### Template 4: CI / testing question

> We added cost gates to our CI pipeline using [AgentGuard](https://github.com/bmdhodl/agent47). It records traces during test runs, then asserts properties like max cost, no loops, and completion time. There's a GitHub Action that fails the build if any assertion breaks.
>
> The EvalSuite API is chainable: `EvalSuite("traces.jsonl").assert_no_loops().assert_budget_under(tokens=50000).run()`

### Template 5: LangGraph specific

> For LangGraph cost tracking, [AgentGuard](https://github.com/bmdhodl/agent47) has a `guarded_node` decorator that wraps any node with budget and loop guards. The budget is shared across all nodes, so a $5 limit applies to the entire graph execution.
>
> You can also add a standalone `guard_node` between steps for explicit budget checks.

## Engagement Rules

1. **Be helpful first.** Only mention AgentGuard when it genuinely solves the problem. Never force it.
2. **No code blocks in comments.** Keep responses short (2-4 sentences), casual, and human. Link to docs for details.
3. **Answer the actual question.** If AgentGuard doesn't solve their specific problem, help anyway. Goodwill compounds.
4. **Never disparage competitors.** State facts about what AgentGuard does. Don't FUD LangSmith, Langfuse, or Portkey.
5. **Disclose when relevant.** If asked directly, say you're the maintainer. Don't hide it.
6. **One comment per thread.** Never reply to yourself or bump. If someone responds, engage naturally.
7. **Track engagement.** Log each post in the tracker below.

## Weekly Tracker

| Week | Date | Channel | Thread | Response | Engagement |
|------|------|---------|--------|----------|------------|
| 1 | | | | | |
| 1 | | | | | |
| 1 | | | | | |

## Metrics (Monthly)

- GitHub stars gained
- PyPI downloads delta
- Dashboard signups
- Inbound GitHub issues from community
- Threads where AgentGuard was mentioned by others (not us)

## Month 1 Targets

- 12 community posts (3/week)
- 5 new GitHub stars
- 50 new PyPI downloads
- 2 dashboard signups
- 1 organic mention by someone else
244 changes: 244 additions & 0 deletions docs/cost-guardrails.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,244 @@
# Cost Guardrails Guide

Stop runaway AI agent costs before they happen. This guide covers everything you need to enforce dollar budgets on agent runs.

## Why Cost Guardrails?

AI agents are expensive and unpredictable. A single agent run can make 3 or 300 LLM calls depending on the task. Without guardrails:

- A stuck agent burns your entire OpenAI budget in minutes
- Cost overruns on autonomous tasks average 340% ([source](https://arxiv.org/abs/2401.15811))
- You only find out when the invoice arrives

AgentGuard's `BudgetGuard` enforces hard dollar limits at runtime — the agent stops the moment it exceeds your budget.

## Quickstart

```bash
pip install agentguard47
```

```python
from agentguard import BudgetGuard, BudgetExceeded

budget = BudgetGuard(max_cost_usd=5.00)

# After each LLM call:
budget.consume(tokens=1500, calls=1, cost_usd=0.045)

# When cumulative cost exceeds $5.00 → BudgetExceeded raised
```

That's it. Three lines to add a hard budget to any agent.

## Configuration

### BudgetGuard Parameters

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `max_tokens` | `int` | `None` | Maximum total tokens. `None` = unlimited |
| `max_calls` | `int` | `None` | Maximum total API calls. `None` = unlimited |
| `max_cost_usd` | `float` | `None` | Maximum total cost in USD. `None` = unlimited |
| `warn_at_pct` | `float` | `None` | Fraction (0.0-1.0) to trigger warning. `None` = no warning |
| `on_warning` | `callable` | `None` | Callback invoked with message when `warn_at_pct` is crossed |

### Examples

```python
# Dollar limit only
BudgetGuard(max_cost_usd=10.00)

# Dollar + call limit with warning
BudgetGuard(max_cost_usd=5.00, max_calls=100, warn_at_pct=0.8)

# Token limit only
BudgetGuard(max_tokens=50_000)

# Full config with warning callback
BudgetGuard(
max_cost_usd=5.00,
max_calls=200,
max_tokens=100_000,
warn_at_pct=0.8,
on_warning=lambda msg: print(f"WARNING: {msg}"),
)
```

### consume() Method

```python
budget.consume(tokens=0, calls=0, cost_usd=0.0)
```

Call after each LLM API call. Pass any combination of tokens, calls, and cost. Raises `BudgetExceeded` if any configured limit is exceeded.

### Checking State

```python
state = budget.state
print(f"Tokens: {state.tokens_used}")
print(f"Calls: {state.calls_used}")
print(f"Cost: ${state.cost_used:.4f}")
```

## How Costs Are Calculated

### Built-in Pricing

AgentGuard includes hardcoded pricing for major models (last updated 2026-02-01):

| Provider | Models |
|----------|--------|
| OpenAI | gpt-4o, gpt-4o-mini, gpt-4-turbo, gpt-4, gpt-3.5-turbo, o1, o1-mini, o3-mini |
| Anthropic | claude-3.5-sonnet, claude-3.5-haiku, claude-3-opus, claude-sonnet-4.5, claude-haiku-4.5, claude-opus-4.6 |
| Google | gemini-1.5-pro, gemini-1.5-flash, gemini-2.0-flash |
| Mistral | mistral-large, mistral-small |
| Meta | llama-3.1-70b |

### Manual Cost Estimation

```python
from agentguard import estimate_cost

cost = estimate_cost("gpt-4o", input_tokens=1000, output_tokens=500)
# → $0.0075
```

### Custom Model Pricing

Add pricing for custom or fine-tuned models:

```python
from agentguard.cost import update_prices

# (input_price_per_1k, output_price_per_1k)
update_prices({("openai", "my-fine-tuned-model"): (0.003, 0.006)})
```

## Auto-Tracking with OpenAI / Anthropic

Skip manual `consume()` calls — patch the SDK to auto-track costs:

```python
from agentguard import Tracer, BudgetGuard, patch_openai, patch_anthropic

tracer = Tracer(
service="my-agent",
guards=[BudgetGuard(max_cost_usd=5.00, warn_at_pct=0.8)],
)

patch_openai(tracer) # auto-tracks all ChatCompletion calls
patch_anthropic(tracer) # auto-tracks all Messages calls

# Use OpenAI/Anthropic normally — costs tracked automatically
```

When done, clean up:

```python
from agentguard import unpatch_openai, unpatch_anthropic

unpatch_openai()
unpatch_anthropic()
```

## LangChain Integration

```bash
pip install agentguard47[langchain]
```

```python
from agentguard import Tracer, BudgetGuard
from agentguard.integrations.langchain import AgentGuardCallbackHandler

tracer = Tracer(service="my-agent")
handler = AgentGuardCallbackHandler(
tracer=tracer,
budget_guard=BudgetGuard(max_cost_usd=5.00),
)

# Pass to any LangChain component
llm = ChatOpenAI(callbacks=[handler])
```

The callback handler auto-extracts token usage from LLM responses and feeds it into BudgetGuard.

## LangGraph Integration

```bash
pip install agentguard47[langgraph]
```

```python
from agentguard import Tracer, BudgetGuard
from agentguard.integrations.langgraph import guarded_node

tracer = Tracer(service="my-graph-agent")
budget = BudgetGuard(max_cost_usd=5.00)

@guarded_node(tracer=tracer, budget_guard=budget)
def research_node(state):
return {"messages": state["messages"] + [result]}
```

## Dashboard Integration

Send traces to the hosted dashboard for centralized monitoring:

```python
from agentguard import Tracer, BudgetGuard, HttpSink

tracer = Tracer(
sink=HttpSink(
url="https://app.agentguard47.com/api/ingest",
api_key="ag_...",
),
guards=[BudgetGuard(max_cost_usd=50.00)],
)
```

The dashboard provides:
- Real-time cost tracking across all agents
- Budget alerts via email and webhook
- Remote kill switch to stop agents mid-run
- Cost breakdown by agent, model, and time period

## CI Cost Gates

Fail CI if agent costs exceed a threshold:

```yaml
- uses: bmdhodl/agent47/.github/actions/agentguard-eval@main
with:
trace-file: traces.jsonl
assertions: "no_errors,max_cost:5.00"
```

Full workflow: [docs/ci/cost-gate-workflow.yml](ci/cost-gate-workflow.yml)

## FAQ

**Q: Does BudgetGuard work without a dashboard?**
Yes. BudgetGuard is local — it runs in your process with zero network calls. The dashboard is optional.

**Q: How accurate are the cost estimates?**
Token-level accurate for supported models. AgentGuard uses published per-token pricing. For models not in the built-in list, use `update_prices()` to add custom pricing.

**Q: What happens when BudgetExceeded is raised?**
It's a normal Python exception. Your agent loop's try/except catches it and you decide what to do — log it, retry with a cheaper model, return a partial result, etc.

**Q: Is it thread-safe?**
Yes. BudgetGuard uses a lock internally. Safe to share across threads.

**Q: Can I reset the budget mid-run?**
Create a new `BudgetGuard` instance. There's no reset method by design — budgets should be immutable per run.

## API Reference

- [`BudgetGuard`](https://github.com/bmdhodl/agent47#guards) — budget enforcement
- [`estimate_cost()`](https://github.com/bmdhodl/agent47#cost-tracking) — per-call cost estimation
- [`patch_openai()`](https://github.com/bmdhodl/agent47#openai--anthropic-auto-instrumentation) — auto-instrumentation
- [`AgentGuardCallbackHandler`](https://github.com/bmdhodl/agent47#langchain) — LangChain integration
- [`guarded_node`](https://github.com/bmdhodl/agent47#langgraph) — LangGraph integration
Loading