Skip to content

Comments

Fix broken multi-turn tool recursion and Anthropic tool result grouping#98

Merged
Godzilla675 merged 3 commits intomainfrom
copilot/fix-broken-ai-functions
Feb 7, 2026
Merged

Fix broken multi-turn tool recursion and Anthropic tool result grouping#98
Godzilla675 merged 3 commits intomainfrom
copilot/fix-broken-ai-functions

Conversation

Copy link
Contributor

Copilot AI commented Feb 7, 2026

handleCopilotMessage had a hand-rolled agent loop that only handled one round of tool calls. After executing a tool and re-calling streamChat, it processed content chunks but silently dropped any subsequent toolCall chunks — breaking multi-step workflows (e.g., search → download → add to timeline). A proper runAgentLoop with depth-limited recursion already existed but was never wired in.

WebSocket handler (handler.ts)

  • Replaced broken inline agent loop with existing runAgentLoop which:
    • Collects all tool calls per turn before executing (parallel tool support)
    • Uses structured toolCalls/toolResults on Message instead of flat strings
    • Recurses up to 10 turns deep

REST API (routes/copilot.ts)

  • Grouped tool results into a single user message (required by Anthropic for parallel tool use):
// Before: each result pushed as separate message — breaks Anthropic
for (const call of toolCalls) {
  messages.push({ role: 'user', content: '...', toolResults: [result] });
}

// After: single grouped message
messages.push({ role: 'user', content: '', toolResults: allResults });
  • Pass executeTool to all loop iterations (was undefined after first call)

Tests

  • websocket-tool-recursion-test.ts — validates 3-turn tool recursion across 2 tool executions
  • rest-tool-grouping-test.ts — validates parallel tool results land in a single message

💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.

Copilot AI and others added 2 commits February 7, 2026 13:34
…on, group tool results for Anthropic compatibility

Co-authored-by: Godzilla675 <131464726+Godzilla675@users.noreply.github.com>
Co-authored-by: Godzilla675 <131464726+Godzilla675@users.noreply.github.com>
Copilot AI changed the title [WIP] Fix broken AI functions for better performance Fix broken multi-turn tool recursion and Anthropic tool result grouping Feb 7, 2026
Copilot AI requested a review from Godzilla675 February 7, 2026 13:39
@Godzilla675 Godzilla675 marked this pull request as ready for review February 7, 2026 16:20
Copilot AI review requested due to automatic review settings February 7, 2026 16:20
@Godzilla675 Godzilla675 merged commit 1851805 into main Feb 7, 2026
4 of 5 checks passed
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR fixes broken multi-turn tool recursion by wiring the WebSocket chat flow to the existing runAgentLoop, and updates the REST /chat loop to group tool results into a single user message (required by Anthropic) while ensuring the tool executor is passed on subsequent iterations.

Changes:

  • WebSocket: replaced the inline one-turn tool handling with runAgentLoop to support multi-turn tool recursion and structured tool messaging.
  • REST: groups parallel tool results into a single toolResults message and passes executeTool to all loop iterations.
  • Added standalone backend test scripts covering WebSocket multi-turn recursion and REST tool-result grouping.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 5 comments.

File Description
apps/backend/src/websocket/handler.ts Swaps in runAgentLoop for WebSocket multi-turn tool recursion.
apps/backend/src/routes/copilot.ts Groups tool results into a single message and preserves executeTool across iterations.
apps/backend/src/test/websocket-tool-recursion-test.ts Adds a script test validating 3-turn recursion with 2 tool executions over WebSocket.
apps/backend/src/test/rest-tool-grouping-test.ts Adds a script test validating tool results are grouped into a single message for REST chat.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines +245 to +247
// Use the agent loop which properly handles multi-turn tool recursion,
// structured messages, and parallel tool calls
await this.runAgentLoop(ws, fullMessages, allTools as any, wsWithHistory, systemPrompt, 0, executeToolWithNotifications, model);
Copy link

Copilot AI Feb 7, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Switching the WebSocket path to runAgentLoop changes tool-result history from a textual user message to a structured toolResults message with content: ''. That breaks multi-turn tool flows for providers that ignore Message.toolResults (notably OpenAIProvider/CustomProvider, which only forward m.content to the API), because the model never sees tool outputs on the next turn and may loop/repeat tool calls. Consider either (a) adding an OpenAI message formatter that maps toolResults to the appropriate OpenAI tool/role messages, or (b) including a textual fallback summary of tool results in content when the active provider doesn't support structured toolResults.

Copilot uses AI. Check for mistakes.
Comment on lines +91 to +95
// Add all tool results as a single user message (content is empty
// because providers use the structured toolResults, not the text field)
messages.push({
role: 'user',
content: '',
Copy link

Copilot AI Feb 7, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This tool-results message is now pushed with content: '' and only structured toolResults. That’s required for Anthropic, but it will break providers that don't serialize toolResults into their API payloads (e.g. OpenAIProvider currently only sends Message.content). To avoid losing tool outputs (and getting stuck in repeated tool calls / hitting MAX_ITERATIONS), either add provider-specific formatting for toolResults (OpenAI tool messages) or include a text fallback summary in content alongside the structured toolResults.

Suggested change
// Add all tool results as a single user message (content is empty
// because providers use the structured toolResults, not the text field)
messages.push({
role: 'user',
content: '',
// Add all tool results as a single user message.
// Keep structured toolResults for providers that support them, and
// also include a text fallback in `content` for providers that only
// use the content field (e.g. some OpenAI-based providers).
messages.push({
role: 'user',
content: toolResults
.map(tr => {
const resultText =
typeof tr.result === 'string'
? tr.result
: JSON.stringify(tr.result);
return `Tool "${tr.toolName}" (callId: ${tr.toolCallId}) returned${tr.isError ? ' error' : ''}: ${resultText}`;
})
.join('\n'),

Copilot uses AI. Check for mistakes.
import { LLMOrchestrator } from '../llm/orchestrator.js';
import { ProjectManager } from '../project/state.js';
import { LLMConfig, Message } from '@ai-video-editor/shared-types';
import { LLMProviderInterface, MCPTool, StreamChunk, ToolCall, ToolExecutor, LLMProviderOptions } from '../llm/types.js';
Copy link

Copilot AI Feb 7, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unused imports LLMProviderOptions, MCPTool, ToolExecutor.

Suggested change
import { LLMProviderInterface, MCPTool, StreamChunk, ToolCall, ToolExecutor, LLMProviderOptions } from '../llm/types.js';
import { LLMProviderInterface, StreamChunk, ToolCall } from '../llm/types.js';

Copilot uses AI. Check for mistakes.
import { ProjectManager } from '../project/state.js';
import { LLMOrchestrator } from '../llm/orchestrator.js';
import { LLMConfig, Message } from '@ai-video-editor/shared-types';
import { LLMProviderInterface, MCPTool, StreamChunk, ToolCall, ToolExecutor, LLMProviderOptions } from '../llm/types.js';
Copy link

Copilot AI Feb 7, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unused imports LLMProviderOptions, MCPTool, ToolExecutor.

Suggested change
import { LLMProviderInterface, MCPTool, StreamChunk, ToolCall, ToolExecutor, LLMProviderOptions } from '../llm/types.js';
import { LLMProviderInterface, StreamChunk, ToolCall } from '../llm/types.js';

Copilot uses AI. Check for mistakes.
const orchestrator = new LLMOrchestrator(mockConfig);
(orchestrator as any).provider = new MockToolProvider();

const wsHandler = new WebSocketHandler(wss, projectManager, orchestrator);
Copy link

Copilot AI Feb 7, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unused variable wsHandler.

Suggested change
const wsHandler = new WebSocketHandler(wss, projectManager, orchestrator);
new WebSocketHandler(wss, projectManager, orchestrator);

Copilot uses AI. Check for mistakes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants