feat(context): sanitize llm context by modalities #4367
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
dosubot
bot
added
size:L
![sourcery-ai[bot]](https://avatars.githubusercontent.com/in/48477?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Hey - 我发现了 2 个问题
给 AI Agent 的提示
Please address the comments from this code review:
## Individual Comments
### Comment 1
<location> `astrbot/core/pipeline/process_stage/method/agent_sub_stages/internal.py:286-287` </location>
<code_context>
+ "tool_calls" in new_msg or "function_call" in new_msg
+ ):
+ new_msg = dict(new_msg)
+ if "tool_calls" in new_msg:
+ removed_tool_calls += 1
+ if "function_call" in new_msg:
+ removed_tool_calls += 1
</code_context>
<issue_to_address>
**nitpick (bug_risk):** 当单条消息中存在多个 tool call 时，tool-call 移除日志会出现低估的情况。
这里的 `removed_tool_calls` 是按键（`tool_calls` / `function_call`）递增，而不是按实际的 tool call 数量递增（例如 `tool_calls` 列表中有多个条目）。如果用于监控，这会让该指标产生误导。可以考虑按 tool call 的数量递增（例如当 `tool_calls` 是列表时使用 `len(new_msg["tool_calls"])`），或者重命名变量，以明确它统计的是“包含 tool call 的消息数”，而不是“单个 tool call 的数量”。
</issue_to_address>
### Comment 2
<location> `astrbot/core/pipeline/process_stage/method/agent_sub_stages/internal.py:209` </location>
<code_context>
)
req.func_tool = None
+ def _sanitize_context_by_modalities(
+ self,
+ provider: Provider,
</code_context>
<issue_to_address>
**issue (complexity):** 建议将 `_sanitize_context_by_modalities` 拆分为更小的辅助函数和简单的数据类，让主循环变成一条简单、易读的流水线。
通过抽取几个小而单一职责的辅助函数，再加上一个简单的统计结构体，在不改变行为的前提下，可以显著降低 `_sanitize_context_by_modalities` 的认知复杂度。
### 1. 抽取能力检测
把 modalities 的解析和能力标记提取到一个辅助函数中。这样可以把主函数中的早期分支噪音移出去：
```python
@dataclass
class ProviderCapabilities:
supports_image: bool
supports_tool_use: bool
def _get_supported_capabilities(self, provider: Provider) -> ProviderCapabilities:
modalities = provider.provider_config.get("modalities")
if not isinstance(modalities, list):
return ProviderCapabilities(supports_image=True, supports_tool_use=True)
normalized = {
str(m).lower() for m in modalities if isinstance(m, str) and m
}
supports_image = bool({"image", "image_url", "vision"} & normalized)
supports_tool_use = bool(
{"tool_use", "tools", "tool", "function_call", "function"} & normalized
)
return ProviderCapabilities(
supports_image=supports_image,
supports_tool_use=supports_tool_use,
)
```
然后在 `_sanitize_context_by_modalities` 的开头：
```python
caps = self._get_supported_capabilities(provider)
if caps.supports_image and caps.supports_tool_use:
return
```
### 2. 将 `is_only_image_placeholder` 抽出
把它做成可复用的辅助函数，而不是内部函数：
```python
PLACEHOLDER_TEXTS = {"[图片]", "[image]", "[Image]", "[IMAGE]"}
def _is_only_image_placeholder(self, parts: list) -> bool:
if not parts:
return False
for part in parts:
if isinstance(part, dict):
if str(part.get("type", "")).lower() != "text":
return False
text = part.get("text", "")
if not isinstance(text, str) or text.strip() not in PLACEHOLDER_TEXTS:
return False
elif isinstance(part, str):
if part.strip() not in PLACEHOLDER_TEXTS:
return False
else:
return False
return True
```
### 3. 使用一个精简的统计对象
用一个专门的结构体来追踪计数器，由各个辅助函数更新，而不是将计数逻辑和业务逻辑交织在一起：
```python
@dataclass
class SanitizeStats:
removed_image_blocks: int = 0
removed_tool_messages: int = 0
removed_tool_calls: int = 0
```
### 4. 将逐条消息的清洗逻辑拆分成辅助函数
让每个辅助函数接收一条消息 + 能力信息 + 统计对象，并返回（可能已修改的）消息或 `None`（表示丢弃）。这样可以把主循环压平成简单流程，去掉嵌套分支。
#### Tool-use 清洗
```python
def _sanitize_tool_use_in_message(
self,
msg: dict,
caps: ProviderCapabilities,
stats: SanitizeStats,
) -> Optional[dict]:
if caps.supports_tool_use:
return msg
role = msg.get("role")
if role == "tool":
stats.removed_tool_messages += 1
return None
if role == "assistant" and ("tool_calls" in msg or "function_call" in msg):
new_msg = dict(msg)
if "tool_calls" in new_msg:
stats.removed_tool_calls += 1
if "function_call" in new_msg:
stats.removed_tool_calls += 1
new_msg.pop("tool_calls", None)
new_msg.pop("function_call", None)
new_msg.pop("tool_call_id", None)
return new_msg
return msg
```
#### 图片清洗
```python
def _sanitize_images_in_message(
self,
msg: dict,
caps: ProviderCapabilities,
stats: SanitizeStats,
) -> Optional[dict]:
if caps.supports_image:
return msg
content = msg.get("content")
if not isinstance(content, list):
return msg
filtered_parts: list = []
removed_any_image = False
for part in content:
if isinstance(part, dict):
part_type = str(part.get("type", "")).lower()
if part_type in {"image_url", "image"}:
removed_any_image = True
stats.removed_image_blocks += 1
continue
filtered_parts.append(part)
if not removed_any_image:
return msg
if not filtered_parts or self._is_only_image_placeholder(filtered_parts):
return None
new_msg = dict(msg)
new_msg["content"] = filtered_parts
return new_msg
```
#### 清理空的 assistant 消息
```python
def _drop_empty_assistant_message(self, msg: dict) -> Optional[dict]:
role = msg.get("role")
if role != "assistant":
return msg
content = msg.get("content")
has_tool_calls = bool(msg.get("tool_calls") or msg.get("function_call"))
if has_tool_calls:
return msg
if content is None:
return None
if isinstance(content, str) and not content.strip():
return None
if isinstance(content, list) and len(content) == 0:
return None
return msg
```
### 5. 简化 `_sanitize_context_by_modalities` 的主循环
在上述辅助函数的基础上，核心逻辑可以变成对每条消息的一条简单流水线：
```python
def _sanitize_context_by_modalities(
self,
provider: Provider,
req: ProviderRequest,
) -> None:
if not self.sanitize_context_by_modalities:
return
if not isinstance(req.contexts, list) or not req.contexts:
return
caps = self._get_supported_capabilities(provider)
if caps.supports_image and caps.supports_tool_use:
return
stats = SanitizeStats()
sanitized_contexts: list[dict] = []
for msg in req.contexts:
if not isinstance(msg, dict):
continue
current = msg
current = self._sanitize_tool_use_in_message(current, caps, stats)
if current is None:
continue
current = self._sanitize_images_in_message(current, caps, stats)
if current is None:
continue
current = self._drop_empty_assistant_message(current)
if current is None:
continue
sanitized_contexts.append(current)
if stats.removed_image_blocks or stats.removed_tool_messages or stats.removed_tool_calls:
logger.debug(
"sanitize_context_by_modalities applied: "
f"removed_image_blocks={stats.removed_image_blocks}, "
f"removed_tool_messages={stats.removed_tool_messages}, "
f"removed_tool_calls={stats.removed_tool_calls}"
)
req.contexts = sanitized_contexts
```
这样在保持所有现有行为的前提下：
- 去除了主循环中的嵌套分支。
- 将关注点拆分开（能力检测、tool 清洗、图片清洗、空 assistant 清理、统计）。
- 让每个辅助函数都可以独立地进行简单的单元测试。
</issue_to_address>帮我变得更有用！请在每条评论上点 👍 或 👎，我会根据你的反馈改进后续的 review。
Original comment in English
Hey - I've found 2 issues
Prompt for AI Agents
Please address the comments from this code review:
## Individual Comments
### Comment 1
<location> `astrbot/core/pipeline/process_stage/method/agent_sub_stages/internal.py:286-287` </location>
<code_context>
+ "tool_calls" in new_msg or "function_call" in new_msg
+ ):
+ new_msg = dict(new_msg)
+ if "tool_calls" in new_msg:
+ removed_tool_calls += 1
+ if "function_call" in new_msg:
+ removed_tool_calls += 1
</code_context>
<issue_to_address>
**nitpick (bug_risk):** Tool-call removal logging undercounts when there are multiple tool calls in a single message.
Here `removed_tool_calls` increments once per key (`tool_calls` / `function_call`), not per actual tool call (e.g., multiple entries in `tool_calls`). This can make the metric misleading if used for monitoring. Consider incrementing by the number of tool calls (e.g., `len(new_msg["tool_calls"])` when it’s a list), or renaming the variable to clarify it counts messages-with-tool-calls, not individual tool calls.
</issue_to_address>
### Comment 2
<location> `astrbot/core/pipeline/process_stage/method/agent_sub_stages/internal.py:209` </location>
<code_context>
)
req.func_tool = None
+ def _sanitize_context_by_modalities(
+ self,
+ provider: Provider,
</code_context>
<issue_to_address>
**issue (complexity):** Consider refactoring `_sanitize_context_by_modalities` into smaller helper functions and simple data classes so the main loop becomes a straightforward, readable pipeline.
You can reduce the cognitive complexity of `_sanitize_context_by_modalities` substantially by extracting a few small, single‑purpose helpers and a tiny stats struct, without changing behavior.
### 1. Extract capability detection
Move the modalities parsing + capability flags into a helper. This removes early branching noise from the main function:
```python
@dataclass
class ProviderCapabilities:
supports_image: bool
supports_tool_use: bool
def _get_supported_capabilities(self, provider: Provider) -> ProviderCapabilities:
modalities = provider.provider_config.get("modalities")
if not isinstance(modalities, list):
return ProviderCapabilities(supports_image=True, supports_tool_use=True)
normalized = {
str(m).lower() for m in modalities if isinstance(m, str) and m
}
supports_image = bool({"image", "image_url", "vision"} & normalized)
supports_tool_use = bool(
{"tool_use", "tools", "tool", "function_call", "function"} & normalized
)
return ProviderCapabilities(
supports_image=supports_image,
supports_tool_use=supports_tool_use,
)
```
Then at the top of `_sanitize_context_by_modalities`:
```python
caps = self._get_supported_capabilities(provider)
if caps.supports_image and caps.supports_tool_use:
return
```
### 2. Move `is_only_image_placeholder` out
Make it a reusable helper instead of an inner function:
```python
PLACEHOLDER_TEXTS = {"[图片]", "[image]", "[Image]", "[IMAGE]"}
def _is_only_image_placeholder(self, parts: list) -> bool:
if not parts:
return False
for part in parts:
if isinstance(part, dict):
if str(part.get("type", "")).lower() != "text":
return False
text = part.get("text", "")
if not isinstance(text, str) or text.strip() not in PLACEHOLDER_TEXTS:
return False
elif isinstance(part, str):
if part.strip() not in PLACEHOLDER_TEXTS:
return False
else:
return False
return True
```
### 3. Use a tiny stats object
Track counters in a dedicated struct that helpers update, instead of interleaving counter manipulation with logic:
```python
@dataclass
class SanitizeStats:
removed_image_blocks: int = 0
removed_tool_messages: int = 0
removed_tool_calls: int = 0
```
### 4. Split per‑message sanitization into helpers
Each helper takes a message + capabilities + stats and either returns a (possibly modified) message or `None` to drop it. This flattens your main loop and removes nested branching.
#### Tool-use sanitization
```python
def _sanitize_tool_use_in_message(
self,
msg: dict,
caps: ProviderCapabilities,
stats: SanitizeStats,
) -> Optional[dict]:
if caps.supports_tool_use:
return msg
role = msg.get("role")
if role == "tool":
stats.removed_tool_messages += 1
return None
if role == "assistant" and ("tool_calls" in msg or "function_call" in msg):
new_msg = dict(msg)
if "tool_calls" in new_msg:
stats.removed_tool_calls += 1
if "function_call" in new_msg:
stats.removed_tool_calls += 1
new_msg.pop("tool_calls", None)
new_msg.pop("function_call", None)
new_msg.pop("tool_call_id", None)
return new_msg
return msg
```
#### Image sanitization
```python
def _sanitize_images_in_message(
self,
msg: dict,
caps: ProviderCapabilities,
stats: SanitizeStats,
) -> Optional[dict]:
if caps.supports_image:
return msg
content = msg.get("content")
if not isinstance(content, list):
return msg
filtered_parts: list = []
removed_any_image = False
for part in content:
if isinstance(part, dict):
part_type = str(part.get("type", "")).lower()
if part_type in {"image_url", "image"}:
removed_any_image = True
stats.removed_image_blocks += 1
continue
filtered_parts.append(part)
if not removed_any_image:
return msg
if not filtered_parts or self._is_only_image_placeholder(filtered_parts):
return None
new_msg = dict(msg)
new_msg["content"] = filtered_parts
return new_msg
```
#### Empty assistant message pruning
```python
def _drop_empty_assistant_message(self, msg: dict) -> Optional[dict]:
role = msg.get("role")
if role != "assistant":
return msg
content = msg.get("content")
has_tool_calls = bool(msg.get("tool_calls") or msg.get("function_call"))
if has_tool_calls:
return msg
if content is None:
return None
if isinstance(content, str) and not content.strip():
return None
if isinstance(content, list) and len(content) == 0:
return None
return msg
```
### 5. Simplify `_sanitize_context_by_modalities` main loop
With the above helpers, the core routine becomes a simple pipeline per message:
```python
def _sanitize_context_by_modalities(
self,
provider: Provider,
req: ProviderRequest,
) -> None:
if not self.sanitize_context_by_modalities:
return
if not isinstance(req.contexts, list) or not req.contexts:
return
caps = self._get_supported_capabilities(provider)
if caps.supports_image and caps.supports_tool_use:
return
stats = SanitizeStats()
sanitized_contexts: list[dict] = []
for msg in req.contexts:
if not isinstance(msg, dict):
continue
current = msg
current = self._sanitize_tool_use_in_message(current, caps, stats)
if current is None:
continue
current = self._sanitize_images_in_message(current, caps, stats)
if current is None:
continue
current = self._drop_empty_assistant_message(current)
if current is None:
continue
sanitized_contexts.append(current)
if stats.removed_image_blocks or stats.removed_tool_messages or stats.removed_tool_calls:
logger.debug(
"sanitize_context_by_modalities applied: "
f"removed_image_blocks={stats.removed_image_blocks}, "
f"removed_tool_messages={stats.removed_tool_messages}, "
f"removed_tool_calls={stats.removed_tool_calls}"
)
req.contexts = sanitized_contexts
```
This preserves all existing behavior while:
- Removing nested branching from the main loop.
- Separating concerns (capabilities, tool sanitization, image sanitization, empty‑assistant pruning, stats).
- Making each helper trivially unit‑testable independently.
</issue_to_address>Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.
Member
|
可以 ruff format 一下，ci 没通过 |
Soulter
changed the title
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Modifications / 改动点
在 provider_settings 中引入了新的配置项 sanitize_context_by_modalities。
（默认开启）开启后，系统会在每次请求 LLM 前，检查当前 Provider 的 modalities 配置。
如果模型不支持 图片 (image, vision)：自动移除历史消息中的图片块。如果模型不支持 工具调用 (tool_use, function_call)：自动移除 tool 角色的消息以及 Assistant 消息中的 tool_calls/function_call 字段。处理了移除内容后可能留下的空消息问题。在 Dashboard 的配置元数据中添加了相关的中英文描述。
Screenshots or Test Results / 运行截图或测试结果
Checklist / 检查清单
requirements.txt和pyproject.toml文件相应位置。/ I have ensured that no new dependencies are introduced, OR if new dependencies are introduced, they have been added to the appropriate locations inrequirements.txtandpyproject.toml.由 Sourcery 提供的摘要
新增一个可选的上下文清理步骤，根据不同服务商的能力，从 LLM 的请求历史中移除不支持的模态内容。
新功能：
sanitize_context_by_modalities，可根据模型配置的模态，自动从请求上下文中移除不受支持的图片内容和工具调用结构。sanitize_context_by_modalities，以便可通过控制台进行配置。增强：
文档：
sanitize_context_by_modalities选项，并提供本地化的说明和提示。Original summary in English
Summary by Sourcery
Add an optional context sanitization step that prunes unsupported modalities from LLM request histories based on provider capabilities.
New Features:
sanitize_context_by_modalitiesto automatically remove unsupported image content and tool-calling structures from request contexts according to the model's configured modalities.sanitize_context_by_modalitiesin default provider settings and the provider template so it can be configured via the dashboard.Enhancements:
Documentation:
sanitize_context_by_modalitiesoption in dashboard configuration metadata with localized descriptions and hints.新特性：
sanitize_context_by_modalities提供方设置，用于根据当前模型所支持的模态，自动从请求上下文中移除不受支持的图片和工具调用结构。sanitize_context_by_modalities选项，以便在控制台中进行配置。改进：
Original summary in English
由 Sourcery 提供的摘要
新增一个可选的上下文清理步骤，根据不同服务商的能力，从 LLM 的请求历史中移除不支持的模态内容。
新功能：
sanitize_context_by_modalities，可根据模型配置的模态，自动从请求上下文中移除不受支持的图片内容和工具调用结构。sanitize_context_by_modalities，以便可通过控制台进行配置。增强：
文档：
sanitize_context_by_modalities选项，并提供本地化的说明和提示。Original summary in English
Summary by Sourcery
Add an optional context sanitization step that prunes unsupported modalities from LLM request histories based on provider capabilities.
New Features:
sanitize_context_by_modalitiesto automatically remove unsupported image content and tool-calling structures from request contexts according to the model's configured modalities.sanitize_context_by_modalitiesin default provider settings and the provider template so it can be configured via the dashboard.Enhancements:
Documentation:
sanitize_context_by_modalitiesoption in dashboard configuration metadata with localized descriptions and hints.