Cache de-serialization does not work on ChatOpenAI with structured output, Pydantic classes and json_schema

[giacbrd]

Checked other resources

• I added a very descriptive title to this issue.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.
• I am sure that this is a bug in LangChain rather than my code.
• The bug is not resolved by updating to the latest stable version of LangChain (or the specific integration package).

Example Code

from langchain.globals import set_llm_cache
from langchain_community.cache import SQLiteCache
from langchain_openai import ChatOpenAI
from pydantic import BaseModel

set_llm_cache(SQLiteCache())

class Temperature(BaseModel):
    value: int
    city: str

chat = ChatOpenAI(model="gpt-4o").with_structured_output(Temperature, method="json_schema")

chat.invoke("What is the average temperature of Rome in May?")
# on this second call the exception is raised
chat.invoke("What is the average temperature of Rome in May?")

Error Message and Stack Trace (if applicable)

Retrieving a cache value that could not be deserialized properly. This is likely due to the cache being in an older format. Please recreate your cache to avoid this error.
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/berardi/Library/Caches/pypoetry/virtualenvs/langchain-elasticsearch-IgW_j1SW-py3.10/lib/python3.10/site-packages/langchain_core/runnables/base.py", line 3020, in invoke
    input = context.run(step.invoke, input, config, **kwargs)
  File "/Users/berardi/Library/Caches/pypoetry/virtualenvs/langchain-elasticsearch-IgW_j1SW-py3.10/lib/python3.10/site-packages/langchain_core/runnables/base.py", line 5352, in invoke
    return self.bound.invoke(
  File "/Users/berardi/Library/Caches/pypoetry/virtualenvs/langchain-elasticsearch-IgW_j1SW-py3.10/lib/python3.10/site-packages/langchain_core/language_models/chat_models.py", line 286, in invoke
    self.generate_prompt(
  File "/Users/berardi/Library/Caches/pypoetry/virtualenvs/langchain-elasticsearch-IgW_j1SW-py3.10/lib/python3.10/site-packages/langchain_core/language_models/chat_models.py", line 786, in generate_prompt
    return self.generate(prompt_messages, stop=stop, callbacks=callbacks, **kwargs)
  File "/Users/berardi/Library/Caches/pypoetry/virtualenvs/langchain-elasticsearch-IgW_j1SW-py3.10/lib/python3.10/site-packages/langchain_core/language_models/chat_models.py", line 643, in generate
    raise e
  File "/Users/berardi/Library/Caches/pypoetry/virtualenvs/langchain-elasticsearch-IgW_j1SW-py3.10/lib/python3.10/site-packages/langchain_core/language_models/chat_models.py", line 633, in generate
    self._generate_with_cache(
  File "/Users/berardi/Library/Caches/pypoetry/virtualenvs/langchain-elasticsearch-IgW_j1SW-py3.10/lib/python3.10/site-packages/langchain_core/language_models/chat_models.py", line 818, in _generate_with_cache
    return ChatResult(generations=cache_val)
  File "/Users/berardi/Library/Caches/pypoetry/virtualenvs/langchain-elasticsearch-IgW_j1SW-py3.10/lib/python3.10/site-packages/pydantic/main.py", line 214, in __init__
    validated_self = self.__pydantic_validator__.validate_python(data, self_instance=self)
pydantic_core._pydantic_core.ValidationError: 1 validation error for ChatResult
generations.0
  Input should be a valid dictionary or instance of ChatGeneration [type=model_type, input_value=Generation(text='{"lc": 1...id_tool_calls": []}}}}'), input_type=Generation]
    For further information visit https://errors.pydantic.dev/2.10/v/model_type

Description

When using the specific configuration in the example, for ChatOpenAI structured output, caches won't work properly.

The problem appears only with Pydantic classes and "json_schema". For example, there is no problem when using TypedDict as schema, or when using "function_calling" as method.

I have also tested Redis and Elasticsearch cache integrations. The issue is the same, LangChain is not able to restore the serialized output from the cache. I am aware of the limits of load methods, the problem however is in the dumping phase, where a Pydantic object is handled by this code. It cannot be properly serialized.

System Info

System Information

OS: Darwin
OS Version: Darwin Kernel Version 24.2.0: Fri Dec 6 18:41:43 PST 2024; root:xnu-11215.61.5~2/RELEASE_X86_64
Python Version: 3.11.7 (main, Jan 11 2024, 20:41:08) [Clang 13.0.0 (clang-1300.0.29.30)]

Package Information

langchain_core: 0.3.24
langchain: 0.3.11
langchain_community: 0.3.11
langsmith: 0.2.3
langchain_elasticsearch: 0.3.0
langchain_openai: 0.2.12
langchain_text_splitters: 0.3.2

Optional packages not installed

langserve

Other Dependencies

aiohttp: 3.11.10
async-timeout: Installed. No version info available.
dataclasses-json: 0.6.7
elasticsearch[vectorstore-mmr]: Installed. No version info available.
httpx: 0.28.1
httpx-sse: 0.4.0
jsonpatch: 1.33
langsmith-pyo3: Installed. No version info available.
numpy: 1.26.4
openai: 1.57.3
orjson: 3.10.12
packaging: 24.2
pydantic: 2.10.3
pydantic-settings: 2.7.0
PyYAML: 6.0.2
requests: 2.32.3
requests-toolbelt: 1.0.0
SQLAlchemy: 2.0.36
tenacity: 9.0.0
tiktoken: 0.8.0
typing-extensions: 4.12.2

[keenborder786]

@giacbrd this happens because of the following line, some objects which are kept as a string in the cache do not implement searlization which in turn raises an exceptions, and returns a Generation. And this Generation object is not compatible with ChatResult which is returned at the following line when chat model hit the cache. The simplest solution is to use InMemoryCache or Redis as cache at the moment as shown:

from langchain.globals import set_llm_cache
from langchain_community.cache import InMemoryCache
from langchain_openai import ChatOpenAI
from pydantic import BaseModel

set_llm_cache(InMemoryCache())

class Temperature(BaseModel):
    value: int
    city: str

chat = ChatOpenAI(model="gpt-4o").with_structured_output(Temperature, method="json_schema")

chat.invoke("What is the average temperature of Rome in May?")
# on this second call the exception is raised
chat.invoke("What is the average temperature of Rome in May?")

[giacbrd]

thanks @keenborder786 , yes I am aware of that limitation, in fact the code referenced in my previous comment is the "back off" of this lack of serialization implementation. From what I understand, it seems a limit in the design of this part. An object must be json serializable or must implement the Serializable interface, so any arbitrary object cannot be properly serialized with the current methods.

Redis cache should have the same problems, according to my tests on a plain Redis server (and also looking at the code). Unfortunately the memory cache, which is a simple unlimited dict, cannot be a choice in a production environment.