chore: detect dynamic model tool support #126
Conversation
Implements Option A from issue lvndry#70 to detect and conditionally pass tools to models based on their capabilities. Changes: - Extended ModelInfo interface with optional supportsTools field - Added supportsTools: true to all static models (OpenAI, Anthropic, etc.) - Enhanced OpenRouter transformer to detect tool support from supported_parameters - Enhanced Ollama transformer with pattern-based tool support detection - Modified ai-sdk-service to conditionally pass tools only when supported - Added logging and warnings when tools are skipped for incompatible models This provides maximum compatibility, allowing users to use any model while gracefully degrading functionality when tools aren't supported. Fixes lvndry#70
…-support Resolved conflicts in: - src/core/constants/models.ts: Took fix/model-detect changes with supportsTools: false for dynamic detection - src/core/types/llm.ts: Made supportsTools required (not optional) - src/services/llm/ai-sdk-service.ts: Used async resolveModelInfo with provider-level caching - src/services/llm/model-fetcher.ts: Used TOOL_PARAMS set and looksToolCapable function for detection
There was a problem hiding this comment.
Most models in this list actually support tools
There was a problem hiding this comment.
I think a better UX would be that if a model does not support tools we should not be able to add tools during creation.
- When user create an agent, if model don't support tools, we should not be able to select tools
- When user create an agent, if mode support tools we should be able to select tools
- When a user edit an agent, if model used to support tools and now doesn't, we clear tools list from config and they should not be able to add tools on edit (tool option should be visible but disabled)
- When a user edit an agent, if model support tools, they should be able to edit tools
That way we always make sure that the agent don't try to call tools they can't execute and users understand they the agent has no tools and thus don't except a different behavior
lvndry
changed the title
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 2 potential issues.
Bugbot Autofix is OFF. To automatically fix reported issues with Cloud Agents, enable Autofix in the Cursor dashboard.
This PR is being reviewed by Cursor Bugbot
Details
You are on the Bugbot Free tier. On this plan, Bugbot will review limited PRs each billing cycle.
To receive Bugbot reviews on all of your PRs, visit the Cursor dashboard to activate Pro and start your 14-day free trial.
src/core/constants/models.ts
Outdated
| id: "deepseek-chat", | ||
| displayName: "DeepSeek Chat", | ||
| isReasoningModel: false, | ||
| supportsTools: false, |
There was a problem hiding this comment.
All static models incorrectly marked as non-tool-capable
High Severity
Every model in STATIC_PROVIDER_MODELS has supportsTools: false, including GPT-4o, Claude, Gemini, Mistral Large, and other models that definitely support function calling. This causes buildToolConfig to disable tools for all these providers, completely breaking tool functionality. The PR reviewer also flagged this: "Most models in this list actually support tools."
| function looksToolCapable(model: OllamaModel): boolean { | ||
| const name = model.name.toLowerCase(); | ||
| if (KNOWN_OLLAMA_TOOL_MODELS.has(model.name)) return true; | ||
| if (name.includes("tool") || name.includes("function")) return true; |
There was a problem hiding this comment.
Ollama tool whitelist check uses wrong case
Low Severity
The looksToolCapable function creates a lowercase version of the model name (name = model.name.toLowerCase()) but then checks the original-cased model.name against KNOWN_OLLAMA_TOOL_MODELS, which contains lowercase strings. The substring checks on the next line correctly use the lowercase name. This inconsistency would cause the whitelist check to fail if Ollama returns model names with any uppercase characters.
* main: chore: reduce bundle size by 50% (lvndry#129) feat: Agent skills (lvndry#127) feat: edit file show diff (lvndry#128) 0.6.2
- Update STATIC_PROVIDER_MODELS to enable tool support for capable models (GPT-4o, Mistral Large, Gemini, etc). - Create Agent: skip tool selection if model does not support tools. - Edit Agent: auto-remove tools if model capabilities change; disable tool editing menu for unsupported models. - Fix Ollama model capability check to be case-insensitive.
daryllundy
force-pushed
the
fix/70-detect-dynamic-model-tool-support
branch
from
bd84d2e to
1e916d8
Compare
| if (currentPredefinedAgent && currentPredefinedAgent.toolCategoryIds.length > 0) { | ||
| await Effect.runPromise( | ||
| terminal.warn( | ||
| `\n⚠️ The selected model (${llmModel}) does not support tools. The "${currentPredefinedAgent.displayName}" agent template's tools will be ignored.`, |
There was a problem hiding this comment.
note sure agent's template is the right name for it
| openai: [ | ||
| { id: "gpt-5.2-pro", displayName: "GPT-5.2 Pro", isReasoningModel: true }, | ||
| { id: "gpt-5.2", displayName: "GPT-5.2", isReasoningModel: true }, | ||
| { id: "gpt-5.2-codex", displayName: "GPT-5.2 Codex", isReasoningModel: true }, | ||
| { id: "gpt-5.1", displayName: "GPT-5.1", isReasoningModel: true }, | ||
| { id: "gpt-5.1-codex", displayName: "GPT-5.1 Codex", isReasoningModel: true }, | ||
| { id: "gpt-5.1-codex-mini", displayName: "GPT-5.1 Codex Mini", isReasoningModel: true }, | ||
| { id: "gpt-5-pro", displayName: "GPT-5 Pro", isReasoningModel: true }, | ||
| { id: "gpt-5", displayName: "GPT-5", isReasoningModel: true }, | ||
| { id: "gpt-5-mini", displayName: "GPT-5 Mini", isReasoningModel: true }, | ||
| { id: "gpt-5-nano", displayName: "GPT-5 Nano", isReasoningModel: true }, | ||
| { id: "gpt-5-codex", displayName: "GPT-5 Codex", isReasoningModel: true }, | ||
| { id: "gpt-4.1", displayName: "GPT-4.1", isReasoningModel: false }, | ||
| { id: "gpt-4.1-mini", displayName: "GPT-4.1 Mini", isReasoningModel: false }, | ||
| { id: "gpt-4.1-nano", displayName: "GPT-4.1 Nano", isReasoningModel: false }, | ||
| { id: "gpt-4o", displayName: "GPT-4o", isReasoningModel: false }, | ||
| { id: "gpt-4o-mini", displayName: "GPT-4o Mini", isReasoningModel: false }, | ||
| { id: "gpt-5.2-pro", displayName: "GPT-5.2 Pro", isReasoningModel: true, supportsTools: false }, | ||
| { id: "gpt-5.2", displayName: "GPT-5.2", isReasoningModel: true, supportsTools: false }, | ||
| { | ||
| id: "gpt-5.2-codex", | ||
| displayName: "GPT-5.2 Codex", | ||
| isReasoningModel: true, | ||
| supportsTools: false, | ||
| }, | ||
| { id: "gpt-5.1", displayName: "GPT-5.1", isReasoningModel: true, supportsTools: false }, | ||
| { | ||
| id: "gpt-5.1-codex", | ||
| displayName: "GPT-5.1 Codex", | ||
| isReasoningModel: true, | ||
| supportsTools: false, | ||
| }, | ||
| { | ||
| id: "gpt-5.1-codex-mini", | ||
| displayName: "GPT-5.1 Codex Mini", | ||
| isReasoningModel: true, | ||
| supportsTools: false, | ||
| }, | ||
| { id: "gpt-5-pro", displayName: "GPT-5 Pro", isReasoningModel: true, supportsTools: false }, | ||
| { id: "gpt-5", displayName: "GPT-5", isReasoningModel: true, supportsTools: false }, | ||
| { id: "gpt-5-mini", displayName: "GPT-5 Mini", isReasoningModel: true, supportsTools: false }, | ||
| { id: "gpt-5-nano", displayName: "GPT-5 Nano", isReasoningModel: true, supportsTools: false }, | ||
| { id: "gpt-5-codex", displayName: "GPT-5 Codex", isReasoningModel: true, supportsTools: false }, |
| id: "magistral-medium-2506", | ||
| displayName: "Magistral Medium", | ||
| isReasoningModel: true, | ||
| supportsTools: false, | ||
| }, | ||
| { | ||
| id: "magistral-small-2506", | ||
| displayName: "Magistral Small", | ||
| isReasoningModel: true, | ||
| supportsTools: false, | ||
| }, |
There was a problem hiding this comment.
please use -latest models + they support tools
| id: model.id, | ||
| displayName: `${model.owned_by.toLowerCase()}/${model.id.toLowerCase()}`, | ||
| isReasoningModel: false, | ||
| supportsTools: false, |
There was a problem hiding this comment.
| function looksToolCapable(model: OllamaModel): boolean { | ||
| const name = model.name.toLowerCase(); | ||
| if (KNOWN_OLLAMA_TOOL_MODELS.has(model.name)) return true; | ||
| if (name.includes("tool") || name.includes("function")) return true; |
| "response_format:json_schema", | ||
| ]); | ||
|
|
||
| const KNOWN_OLLAMA_TOOL_MODELS = new Set<string>([ |
There was a problem hiding this comment.
const toolModels = [
"nemotron-3-nano",
"functiongemma",
"devstral-small-2",
"devstral-2",
"ministral-3",
"qwen3-vl",
"gpt-oss",
"deepseek-r1",
"qwen3-coder",
"llama3.1",
"llama3.2",
"mistral",
"qwen2.5",
"qwen3",
"qwen2.5-coder",
"qwen2",
"mistral-nemo",
"llama3.3",
"smollm2",
"mistral-small",
"granite3.1-moe",
"qwq",
"mixtral",
"cogito",
"mistral-small3.2",
"llama4",
"magistral",
"granite3.3",
"phi4-mini",
"granite3.2-vision",
"devstral",
"command-r",
"granite4",
"mistral-small3.1",
"hermes3",
"mistral-large",
"command-r-plus",
"granite3-dense",
"granite3.1-dense",
"deepseek-v3.1",
"qwen3-next",
"llama3-groq-tool-use",
"aya-expanse",
"granite3-moe",
"granite3.2",
"nemotron-mini",
"athene-v2",
"nemotron",
"firefunction-v2",
"command-r7b",
"command-a",
"command-r7b-arabic",
"olmo-3.1",
"gpt-oss-safeguard",
"rnj-1",
"glm-4.7-flash",
];
|
|
||
| function looksToolCapable(model: OllamaModel): boolean { | ||
| const name = model.name.toLowerCase(); | ||
| if (KNOWN_OLLAMA_TOOL_MODELS.has(name)) return true; |
There was a problem hiding this comment.
if (KNOWN_OLLAMA_TOOL_MODELS.find((name_prefix) => name.startsWith(name_prefix)))
3 participants
Description
This change updates Jazz's LLM integration to detect tool support on a per‑model basis and only enable tools when the selected model actually supports them. This prevents tool calls from being sent to chat‑only models and surfaces a clear signal back through the agent pipeline when tools are disabled.
Concretely:
ModelInfowith asupportsToolsflag and wire it through static and dynamically fetched provider models.buildToolConfighelper in the AI SDK service to:toolsDisabledflag intoChatCompletionResponseandAgentResponse, so the agent layer and UI can inform users when tools were skipped.This aligns Jazz's behavior with providers that distinguish between tool‑capable and chat‑only models and avoids confusing "silent failures" when tools are configured but unusable.
Type of Change
How it works
STATIC_PROVIDER_MODELS, each entry now declaressupportsTools, defaulting tofalseunless explicitly enabled.ModelFetcherServicenormalizes provider metadata intoModelInfoand setssupportsToolsbased on each provider's capabilities.AISDKServiceresolves the selected model'sModelInfo, callsbuildToolConfig, and:toolsmap only when the model supports tools.toolChoiceonly when tools are allowed.toolsDisabled: truetoAgentResponsewhen tools were skipped, enabling the UI to show a "tools disabled for this model" hint instead of silently failing tool calls.Testing
toolsDisabledremainsfalse.toolsDisabledis set onChatCompletionResponseandAgentResponse.supportsToolsfield.Checklist
Note
Introduces per-model tool capability detection and propagates a clear signal when tools are skipped.
ModelInfowithsupportsTools; updates static model lists and dynamic fetchers (OpenRouter, Ollama) to populate it; caches fetched model metadata per providerbuildToolConfigandtoAISDKToolChoicein AI SDK service to omittools/toolChoicefor non-tool models and recordtoolsDisabledtoolsDisabledthroughChatCompletionResponseandAgentResponse; batch/streaming executors attach it to responses; stream processor includes it in final eventsWritten by Cursor Bugbot for commit bd84d2e. This will update automatically on new commits. Configure here.