Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions docs/source/ko/_toctree.yml
Original file line number Diff line number Diff line change
Expand Up @@ -10,8 +10,8 @@
sections:
- local: tutorials/building_good_agents
title: ์ข‹์€ ์—์ด์ „ํŠธ ๊ตฌ์ถ•ํ•˜๊ธฐ
# - local: tutorials/inspect_runs
# title: ๐Ÿ“Š Inspect your agent runs using telemetry
- local: tutorials/inspect_runs
title: ๐Ÿ“Š ํ…”๋ ˆ๋ฉ”ํŠธ๋ฆฌ๋กœ ์—์ด์ „ํŠธ ์‹คํ–‰ ๊ฒ€์‚ฌํ•˜๊ธฐ
# - local: tutorials/tools
# title: ๐Ÿ› ๏ธ Tools - in-depth guide
# - local: tutorials/secure_code_execution
Expand Down
179 changes: 179 additions & 0 deletions docs/source/ko/tutorials/inspect_runs.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,179 @@
# OpenTelemetry๋กœ ์‹คํ–‰ ๊ฒ€์‚ฌํ•˜๊ธฐ[[inspecting-runs-with-opentelemetry]]

[[open-in-colab]]

> [!TIP]
> ์—์ด์ „ํŠธ ๊ตฌ์ถ•์ด ์ฒ˜์Œ์ด๋ผ๋ฉด ๋จผ์ € [์—์ด์ „ํŠธ ์†Œ๊ฐœ](../conceptual_guides/intro_agents)์™€ [์•ˆ๋‚ด์„œ](../guided_tour)๋ฅผ ์ฝ์–ด๋ณด์„ธ์š”.

## ์—์ด์ „ํŠธ ์‹คํ–‰์„ ๋กœ๊น…ํ•˜๋Š” ์ด์œ ๋Š”?[[why-log-your-agent-runs?]]

์—์ด์ „ํŠธ ์‹คํ–‰์„ ๋””๋ฒ„๊น…ํ•˜๋Š” ๊ฒƒ์€ ๋ณต์žกํ•œ ์ž‘์—…์ž…๋‹ˆ๋‹ค.

์‹คํ–‰์ด ์ œ๋Œ€๋กœ ์ง„ํ–‰๋˜์—ˆ๋Š”์ง€ ํ™•์ธํ•˜๊ธฐ ์–ด๋ ต์Šต๋‹ˆ๋‹ค. ์—์ด์ „ํŠธ ์›Œํฌํ”Œ๋กœ์šฐ๋Š” ์„ค๊ณ„์ƒ ์˜ˆ์ธก ๋ถˆ๊ฐ€๋Šฅํ•˜๊ธฐ ๋•Œ๋ฌธ์ž…๋‹ˆ๋‹ค(๋งŒ์•ฝ ์˜ˆ์ธก ๊ฐ€๋Šฅํ–ˆ๋‹ค๋ฉด ์ผ๋ฐ˜์ ์ธ ์ฝ”๋“œ๋ฅผ ์‚ฌ์šฉํ–ˆ์„ ๊ฒƒ์ž…๋‹ˆ๋‹ค).

์‹คํ–‰ ๊ณผ์ •์„ ์‚ดํŽด๋ณด๋Š” ๊ฒƒ๋„ ์‰ฝ์ง€ ์•Š์Šต๋‹ˆ๋‹ค. ๋‹ค๋‹จ๊ณ„ ์—์ด์ „ํŠธ๋Š” ์ฝ˜์†”์„ ๋กœ๊ทธ๋กœ ๋น ๋ฅด๊ฒŒ ์ฑ„์šฐ๋Š” ๊ฒฝํ–ฅ์ด ์žˆ์œผ๋ฉฐ, ๋Œ€๋ถ€๋ถ„์˜ ์˜ค๋ฅ˜๋Š” ๋‹จ์ˆœํ•œ "LLM์˜ ์‹ค์ˆ˜" ์œ ํ˜•์œผ๋กœ, LLM์ด ๋‹ค์Œ ๋‹จ๊ณ„์—์„œ ๋” ๋‚˜์€ ์ฝ”๋“œ๋‚˜ ๋„๊ตฌ ํ˜ธ์ถœ์„ ์ž‘์„ฑํ•˜์—ฌ ์Šค์Šค๋กœ ๊ต์ •ํ•ฉ๋‹ˆ๋‹ค.

๋”ฐ๋ผ์„œ ๋‚˜์ค‘์— ๊ฒ€์‚ฌํ•˜๊ณ  ๋ชจ๋‹ˆํ„ฐ๋งํ•  ์ˆ˜ ์žˆ๋„๋ก ๊ณ„์ธก์„ ํ†ตํ•ด ์—์ด์ „ํŠธ ์‹คํ–‰์„ ๊ธฐ๋กํ•˜๋Š” ๊ฒƒ์ด ํ”„๋กœ๋•์…˜ ํ™˜๊ฒฝ์—์„œ๋Š” ํ•„์ˆ˜์ž…๋‹ˆ๋‹ค!

์—์ด์ „ํŠธ ์‹คํ–‰์„ ๊ณ„์ธกํ•˜๊ธฐ ์œ„ํ•ด [OpenTelemetry](https://opentelemetry.io/) ํ‘œ์ค€์„ ๋„์ž…ํ–ˆ์Šต๋‹ˆ๋‹ค.

์ฆ‰, ๊ณ„์ธก ์ฝ”๋“œ๋ฅผ ์‹คํ–‰ํ•œ ํ›„ ์—์ด์ „ํŠธ๋ฅผ ํ‰์†Œ์ฒ˜๋Ÿผ ์‹คํ–‰ํ•˜๋ฉด ๋ชจ๋“  ๋‚ด์šฉ์ด ํ”Œ๋žซํผ์— ์ž๋™์œผ๋กœ ๋กœ๊น…๋ฉ๋‹ˆ๋‹ค. ๋‹ค์–‘ํ•œ OpenTelemetry ๋ฐฑ์—”๋“œ์—์„œ ์ด๋ฅผ ๊ตฌํ˜„ํ•˜๋Š” ๋ฐฉ๋ฒ•์˜ ์˜ˆ์‹œ๋ฅผ ์•„๋ž˜์— ์ œ์‹œํ•ฉ๋‹ˆ๋‹ค.

ํ”Œ๋žซํผ์—์„œ์˜ ์‹ค์ œ ๋ชจ์Šต์€ ๋‹ค์Œ๊ณผ ๊ฐ™์Šต๋‹ˆ๋‹ค.

<div class="flex justify-center">
<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/smolagents/inspect_run_phoenix.gif"/>
</div>

## Arize AI Phoenix๋กœ ํ…”๋ ˆ๋ฉ”ํŠธ๋ฆฌ ์„ค์ •[[setting-up-telemetry-with-arize-ai-phoenix]]

๋จผ์ € ํ•„์š”ํ•œ ํŒจํ‚ค์ง€๋ฅผ ์„ค์น˜ํ•ฉ๋‹ˆ๋‹ค. ์—ฌ๊ธฐ์„œ๋Š” ๋กœ๊ทธ๋ฅผ ์ˆ˜์ง‘ํ•˜๊ณ  ๊ฒ€์‚ฌํ•˜๊ธฐ์— ์ข‹์€ ์†”๋ฃจ์…˜์ธ [Arize AI์˜ Phoenix](https://github.com/Arize-ai/phoenix)๋ฅผ ์„ค์น˜ํ•˜์ง€๋งŒ, ์ด ๊ณผ์ •์—๋Š” ๋‹ค๋ฅธ OpenTelemetry ํ˜ธํ™˜ ํ”Œ๋žซํผ์„ ํ™œ์šฉํ•  ์ˆ˜๋„ ์žˆ์Šต๋‹ˆ๋‹ค.

```shell
pip install 'smolagents[telemetry,toolkit]'
```

๋‹ค์Œ ๋‹จ๊ณ„๋กœ ์ˆ˜์ง‘๊ธฐ๋ฅผ ๋ฐฑ๊ทธ๋ผ์šด๋“œ์—์„œ ์‹คํ–‰ํ•ฉ๋‹ˆ๋‹ค.

```shell
python -m phoenix.server.main serve
```

๋งˆ์ง€๋ง‰์œผ๋กœ `SmolagentsInstrumentor`๋ฅผ ์„ค์ •ํ•˜์—ฌ ์—์ด์ „ํŠธ๋ฅผ ์ถ”์ ํ•˜๊ณ  Phoenix ๊ธฐ๋ณธ ์—”๋“œํฌ์ธํŠธ๋กœ ํ•ด๋‹น ์ถ”์  ๋ฐ์ดํ„ฐ๋ฅผ ์ „์†กํ•ฉ๋‹ˆ๋‹ค.

```python
from phoenix.otel import register
from openinference.instrumentation.smolagents import SmolagentsInstrumentor

register()
SmolagentsInstrumentor().instrument()
```
์ด์ œ ์—์ด์ „ํŠธ๋ฅผ ์‹คํ–‰ํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค!

```py
from smolagents import (
CodeAgent,
ToolCallingAgent,
WebSearchTool,
VisitWebpageTool,
InferenceClientModel,
)

model = InferenceClientModel()

search_agent = ToolCallingAgent(
tools=[WebSearchTool(), VisitWebpageTool()],
model=model,
name="search_agent",
description="This is an agent that can do web search.",
)

manager_agent = CodeAgent(
tools=[],
model=model,
managed_agents=[search_agent],
)
manager_agent.run(
"If the US keeps its 2024 growth rate, how many years will it take for the GDP to double?"
)
```
๋์ž…๋‹ˆ๋‹ค!
์ด์ œ `http://0.0.0.0:6006/projects/`๋กœ ์ด๋™ํ•˜์—ฌ ์‹คํ–‰ ๊ฒฐ๊ณผ๋ฅผ ํ™•์ธํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค!

<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/smolagents/inspect_run_phoenix.png">

CodeAgent๊ฐ€ ๊ด€๋ฆฌํ•˜๋Š” ToolCallingAgent๋ฅผ ํ˜ธ์ถœํ•˜์—ฌ(์ฐธ๊ณ ๋กœ ๊ด€๋ฆฌ๋˜๋Š” ์—์ด์ „ํŠธ๋Š” CodeAgent๊ฐ€ ๋  ์ˆ˜๋„ ์žˆ์Šต๋‹ˆ๋‹ค) ๋ฏธ๊ตญ 2024๋…„ ์„ฑ์žฅ๋ฅ ์„ ์›น์—์„œ ๊ฒ€์ƒ‰ํ•˜๋„๋ก ์š”์ฒญํ•œ ๊ฒƒ์„ ํ™•์ธํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. ์ดํ›„ ๊ด€๋ฆฌ๋˜๋Š” ์—์ด์ „ํŠธ๊ฐ€ ๊ฒฐ๊ณผ๋ฅผ ๋ณด๊ณ ํ•˜๋ฉด, ๊ด€๋ฆฌ์ž ์—์ด์ „ํŠธ๊ฐ€ ์ด ์ •๋ณด๋ฅผ ํ™œ์šฉํ•˜์—ฌ ๊ฒฝ์ œ ๋ฐฐ์ฆ ์‹œ๊ฐ„์„ ๊ณ„์‚ฐํ–ˆ์Šต๋‹ˆ๋‹ค! ํฅ๋ฏธ๋กญ์ฃ ?

## ๐Ÿชข Langfuse๋กœ ํ…”๋ ˆ๋ฉ”ํŠธ๋ฆฌ ์„ค์ •[[setting-up-telemetry-with-๐Ÿชข-langfuse]]

์ด ๋ถ€๋ถ„์€ `SmolagentsInstrumentor`๋ฅผ ์‚ฌ์šฉํ•˜์—ฌ **Langfuse**๋กœ Hugging Face **smolagents**๋ฅผ ๋ชจ๋‹ˆํ„ฐ๋งํ•˜๊ณ  ๋””๋ฒ„๊น…ํ•˜๋Š” ๋ฐฉ๋ฒ•์„ ๋ณด์—ฌ์ค๋‹ˆ๋‹ค.

> **Langfuse๋ž€?** [Langfuse](https://langfuse.com)๋Š” LLM ์—”์ง€๋‹ˆ์–ด๋ง์„ ์œ„ํ•œ ์˜คํ”ˆ์†Œ์Šค ํ”Œ๋žซํผ์ž…๋‹ˆ๋‹ค. AI ์—์ด์ „ํŠธ๋ฅผ ์œ„ํ•œ ์ถ”์  ๋ฐ ๋ชจ๋‹ˆํ„ฐ๋ง ๊ธฐ๋Šฅ์„ ์ œ๊ณตํ•˜์—ฌ ๊ฐœ๋ฐœ์ž๊ฐ€ ์ œํ’ˆ์„ ๋””๋ฒ„๊น…ํ•˜๊ณ , ๋ถ„์„ํ•˜๊ณ , ์ตœ์ ํ™”ํ•  ์ˆ˜ ์žˆ๋„๋ก ๋„์™€์ค๋‹ˆ๋‹ค. Langfuse๋Š” ๋„ค์ดํ‹ฐ๋ธŒ ํ†ตํ•ฉ, OpenTelemetry, SDK๋ฅผ ํ†ตํ•ด ๋‹ค์–‘ํ•œ ๋„๊ตฌ์™€ ํ”„๋ ˆ์ž„์›Œํฌ์™€ ํ†ตํ•ฉ๋ฉ๋‹ˆ๋‹ค.

### 1๋‹จ๊ณ„: ์˜์กด์„ฑ ์„ค์น˜[[step-1:-install-dependencies]]

```python
%pip install langfuse 'smolagents[telemetry]' openinference-instrumentation-smolagents
```

### 2๋‹จ๊ณ„: ํ™˜๊ฒฝ ๋ณ€์ˆ˜ ์„ค์ •[[step-2:-set-up-environment-variables]]

Langfuse API ํ‚ค๋ฅผ ์„ค์ •ํ•˜๊ณ  Langfuse๋กœ ์ถ”์ ์„ ๋ณด๋‚ด๋„๋ก OpenTelemetry ์—”๋“œํฌ์ธํŠธ๋ฅผ ๊ตฌ์„ฑํ•˜์„ธ์š”. [Langfuse Cloud](https://cloud.langfuse.com)์— ๊ฐ€์ž…ํ•˜๊ฑฐ๋‚˜ [Langfuse๋ฅผ ์ž์ฒด ํ˜ธ์ŠคํŒ…](https://langfuse.com/self-hosting)ํ•˜์—ฌ Langfuse API ํ‚ค๋ฅผ ์–ป์œผ์„ธ์š”.

๋˜ํ•œ [Hugging Face ํ† ํฐ](https://huggingface.co/settings/tokens) (`HF_TOKEN`)์„ ํ™˜๊ฒฝ ๋ณ€์ˆ˜๋กœ ์ถ”๊ฐ€ํ•˜์„ธ์š”.

```python
import os
# ํ”„๋กœ์ ํŠธ ์„ค์ • ํŽ˜์ด์ง€(https://cloud.langfuse.com)์—์„œ ํ”„๋กœ์ ํŠธ ํ‚ค๋ฅผ ๊ฐ€์ ธ์˜ต๋‹ˆ๋‹ค.
os.environ["LANGFUSE_PUBLIC_KEY"] = "pk-lf-..."
os.environ["LANGFUSE_SECRET_KEY"] = "sk-lf-..."
os.environ["LANGFUSE_HOST"] = "https://cloud.langfuse.com" # ๐Ÿ‡ช๐Ÿ‡บ ์œ ๋Ÿฝ ์ง€์—ญ
# os.environ["LANGFUSE_HOST"] = "https://us.cloud.langfuse.com" # ๐Ÿ‡บ๐Ÿ‡ธ ๋ฏธ๊ตญ ์ง€์—ญ

# Hugging Face ํ† ํฐ์„ ์ž…๋ ฅํ•ฉ๋‹ˆ๋‹ค.
os.environ["HF_TOKEN"] = "hf_..."
```

ํ™˜๊ฒฝ ๋ณ€์ˆ˜๊ฐ€ ์„ค์ •๋˜๋ฉด ์ด์ œ Langfuse ํด๋ผ์ด์–ธํŠธ๋ฅผ ์ดˆ๊ธฐํ™”ํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. `get_client()`๋Š” ํ™˜๊ฒฝ ๋ณ€์ˆ˜์— ์ œ๊ณต๋œ ์ž๊ฒฉ ์ฆ๋ช…์„ ์‚ฌ์šฉํ•˜์—ฌ Langfuse ํด๋ผ์ด์–ธํŠธ๋ฅผ ์ดˆ๊ธฐํ™”ํ•ฉ๋‹ˆ๋‹ค.

```python
from langfuse import get_client

langfuse = get_client()

# ์—ฐ๊ฒฐ์„ ํ™•์ธํ•ฉ๋‹ˆ๋‹ค.
if langfuse.auth_check():
print("Langfuse client is authenticated and ready!")
else:
print("Authentication failed. Please check your credentials and host.")
```

### 3๋‹จ๊ณ„: `SmolagentsInstrumentor` ์ดˆ๊ธฐํ™”[[step-3:-initialize-the-`smolagentsinstrumentor`]]

์• ํ”Œ๋ฆฌ์ผ€์ด์…˜ ์ฝ”๋“œ๋ฅผ ์‹คํ–‰ํ•˜๊ธฐ ์ „์— `SmolagentsInstrumentor`๋ฅผ ์ดˆ๊ธฐํ™”ํ•˜์„ธ์š”.

```python
from openinference.instrumentation.smolagents import SmolagentsInstrumentor

SmolagentsInstrumentor().instrument()
```

### 4๋‹จ๊ณ„: smolagent ์‹คํ–‰[[step-4:-run-your-smolagent]]

```python
from smolagents import (
CodeAgent,
ToolCallingAgent,
WebSearchTool,
VisitWebpageTool,
InferenceClientModel,
)

model = InferenceClientModel(
model_id="deepseek-ai/DeepSeek-R1-Distill-Qwen-32B"
)

search_agent = ToolCallingAgent(
tools=[WebSearchTool(), VisitWebpageTool()],
model=model,
name="search_agent",
description="This is an agent that can do web search.",
)

manager_agent = CodeAgent(
tools=[],
model=model,
managed_agents=[search_agent],
)
manager_agent.run(
"How can Langfuse be used to monitor and improve the reasoning and decision-making of smolagents when they execute multi-step tasks, like dynamically adjusting a recipe based on user feedback or available ingredients?"
)
```

### 5๋‹จ๊ณ„: Langfuse์—์„œ ์ถ”์  ๋ณด๊ธฐ[[step-5:-view-traces-in-langfuse]]

์—์ด์ „ํŠธ๋ฅผ ์‹คํ–‰ํ•œ ํ›„, Langfuse์˜ smolagents ์• ํ”Œ๋ฆฌ์ผ€์ด์…˜์—์„œ ์ƒ์„ฑ๋œ ์ถ”์  ์ •๋ณด๋ฅผ ํ™•์ธํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. AI ์—์ด์ „ํŠธ์˜ ๋””๋ฒ„๊น…๊ณผ ์ตœ์ ํ™”์— ๋„์›€์ด ๋˜๋Š” LLM ์ƒํ˜ธ์ž‘์šฉ์˜ ์ƒ์„ธํ•œ ์„ธ๋ถ€ ๊ณผ์ •์„ ์‚ดํŽด๋ณผ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

![smolagents example trace](https://langfuse.com/images/cookbook/integration-smolagents/smolagent_example_trace.png)

_[Langfuse์˜ ์ถ”์  ์˜ˆ์‹œ](https://cloud.langfuse.com/project/cloramnkj0002jz088vzn1ja4/traces/ce5160f9bfd5a6cd63b07d2bfcec6f54?timestamp=2025-02-11T09%3A25%3A45.163Z&display=details)_