[Bug]: json mode with nested models doesn't work for Anthropic

[amanpreet692]

What happened?

A bug happened!
Picking the example from your docs @ https://docs.litellm.ai/docs/completion/json_mode:

import os
from litellm import completion 
from pydantic import BaseModel

# add to env var 
os.environ["OPENAI_API_KEY"] = ""

messages = [{"role": "user", "content": "List 5 important events in the XIX century"}]

class CalendarEvent(BaseModel):
  name: str
  date: str
  participants: list[str]

class EventsList(BaseModel):
    events: list[CalendarEvent]

resp = completion(
    model="gpt-4o-2024-08-06",
    messages=messages,
    response_format=EventsList
)

print("Received={}".format(resp))

The above won't work if you replace model with anthropic/claude-3-5-sonnet-20241022 and add the anthropic key.

Relevant log output

**Error** 

``litellm.exceptions.BadRequestError: litellm.BadRequestError: AnthropicException - {"type":"error","error":{"type":"invalid_request_error","message":"tools.0.input_schema: JSON schema is invalid - please consult https://json-schema.org or our documentation at https://docs.anthropic.com/en/docs/tool-use"}}``

Are you a ML Ops Team?

Yes

What LiteLLM version are you on ?

v1.53.1 - v1.58.0

Twitter / LinkedIn details

No response

[sava-t]

Yes! I've been having similar problems.

[krrishdholakia]

able to repro

[krrishdholakia]

So i can see a valid json schema being generated, but anthropic seems to reject it. Will need to investigate further.

You can see the raw request to anthropic with litellm._turn_on_debug()

Any help analyzing this further would be great! (e.g. how should the schema for anthropic be handled differently)

relevant code - anthropic -

litellm/litellm/llms/anthropic/chat/transformation.py

Line 374 in df7d500

_input_schema["properties"] = {"values": json_schema}

convert pydantic to json schema -

litellm/litellm/utils.py

Line 2777 in df7d500

non_default_params["response_format"] = type_to_response_format_param(

[amanpreet692]

Thanks for the pointers @krrishdholakia !
I tried a few things and this snippet works:

messages = [{"role": "user", "content": "List 5 important events in the XIX century"}]


class CalendarEvent(BaseModel):
    name: str
    date: str
    participants: list[str]


class EventsList(BaseModel):
    events: list[CalendarEvent]


schema = EventsList.model_json_schema(ref_template="/$defs/{model}")
json_schema = {
    "type": "json_schema",
    "json_schema": {
        "schema": schema,
        "name": EventsList.__name__,
        "strict": True,
    }
}
resp = litellm.completion(
    model="anthropic/claude-3-5-sonnet-20241022",
    messages=messages,
    response_format=json_schema)

res_str = resp["choices"][0]["message"]["content"].strip()
print(json.loads(res_str))

Somehow Anthropic has an issue with the default ref template of "#/$defs/{model}" . It doesn't process the pound sign and errors out while processing the refs in case of nested jsons.
It could potentially be because the json schema is first processed through the open ai _pydantic.py script which might not be compatible with anthropic but I can't be sure about this.

Hope this was helpful!

[krrishdholakia]

Amazing, thank you for the find. I'll push a fix today. @amanpreet692