Expected behavior of describe method

[ariel-pettyjohn]

I'm trying to understand the expected behavior of Zod's describe method when used with Genkit.

Judging by the Genkit document, my expectation was that it was a utility to pass additional context to the model; however, the Zod documentation leads me to expect that it's intended to add a description field to the schema.

I've seen Genkit use describe to do the latter, which I initially thought was a bug, so I created this issue; however, I closed it to avoid becoming clutter after I reviewed the Zod documentation.

The remaining problem, which you can see in the example in the original issue, is that descriptions from child schema seem to "bubble-up" or "leak" into their parent schema. In my example, instead of a color's name, hex, or rgb fields receiving the given description, the color itself receives a null description.

This still seems like a bug to me at least, because the behavior is inconsistent either way, but I'm not sure how to properly characterize and report it until I understand the expected behavior.

What's the expected behavior in this case, or is it just user error on my part?

Thanks in advance!

[mbleigh]

A quick aside on how output conformance for Genkit works: we take the provided Zod schema, convert it to JSON schema, then augment the prompt with instructions to format output as conforming to the JSON schema. When you compose a Zod schema from multiple subschemas, they are all converged into one large schema with all nested fields.

With that context in mind and taking a look at your reported issue, I think there are a few things at play:

1. You are using the Gemini 1.0 Pro model which is fairly poor at output conformance - upgrading to Gemini 1.5 Flash may significantly improve outcomes.
2. I'd recommend fully describing the values you expect in natural language -- e.g. the color's RGB value as an array of integers from 0 to 255 e.g. [100,255,0]. I'm not certain to what extend our conversion of Zod to JSON schema captures constraints like regex validations or ranges.

In general when you're seeing poor output conformance results a good tactic is to add good examples in the prompt:

=== Examples of color extraction

Input: the quick red fox jumped over the lazy red dog
Output: {colors: [{name: "red"}, {name: "brown"}]

^ obviously the above should be tailored for your use case. If you add some diverse examples with correct output, the LLM will oftentimes behave better.

Hope that helps!

[ariel-pettyjohn]

Thanks @mbleigh! That's great advice going forward 👍

I've been going back and forth between using 1.0 and 1.5 to compare results and test, so I'll make sure to double-check with 1.5 first out of curiosity.

Now that you mention it, I wonder if adding examples, and more explicit descriptions, won't also improve results when trying to convert color formats as well, which initially seemed to confuse both 1.0 and 1.5.

Just to speculate for a minute, I've looked into the code a little and I'm a bit surprised that the model wouldn't just fail in cases like these, at least from what I saw in schema.ts.

For example: at first I wondered whether I might've confused the model with unclear and inadvertently self-referential descriptions, say by specifying "colors" to contain hex values in such a way that it then interprets those hex "color values" to be "colors" themselves, which in turn contain further hex values and so on.

On the other hand, I also assume the JSON validator would just throw an error if the model actually tried to produce a cyclical output schema, at least from what I've seen of the code.

Since I didn't describe the color schema itself, only its subschema (subschemata?), this looks like a "leaky" output to me because the descriptions seem to have "bubbled up" from the subschema to their parent, so I thought something circular might be at fault.

From the perspective of explainability/interpretability, do we know what's behind this apparent "leakiness", or maybe Zod just uses some kind of collapse/fallback logic I don't know about?

At a high level, with a description, I guess we are venturing into the realm of semantic verification vs. syntactic validation of the schema, so maybe this is just to be expected from models like 1.0?

Thanks again for all your time and hard work! 🙏