Fix structured outputs on various models

[dirkbrnd]

Description

This addresses structured output not behaving exactly correct on multiple models

fixes: #2099

---

Type of change

Please check the options that are relevant:

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Model update (Addition or modification of models)
• Other (please describe):

---

Checklist

• Adherence to standards: Code complies with Agno’s style guidelines and best practices.
• Formatting and validation: You have run ./scripts/format.sh and ./scripts/validate.sh to ensure code is formatted and linted.
• Self-review completed: A thorough review has been performed by the contributor(s).
• Documentation: Docstrings and comments have been added or updated for any complex logic.
• Examples and guides: Relevant cookbook examples have been included or updated (if applicable).
• Tested in a clean environment: Changes have been tested in a clean environment to confirm expected behavior.
• Tests (optional): Tests have been added or updated to cover any new or changed functionality.

---

Additional Notes

Include any deployment notes, performance implications, security considerations, or other relevant information (e.g., screenshots or logs if applicable).

[ysolanky]

The response_model should be set at the Agent level. We should make sure it is consistent for each provider

[dirkbrnd]

What goes to the model is only response_format and whether to use structured_output or not. We don't send the field response_model, that is only used in the agent.

[ysolanky]

Looks like we are just defining this field and not using it anywhere

[dirkbrnd]

This logic in the agent is the important part.

# Update the response_format on the Model
        if self.response_model is not None:
            # This will pass the pydantic model to the model
            if self.structured_outputs and self.model.supports_structured_outputs:
                logger.debug("Setting Model.response_format to Agent.response_model")
                self.model.response_format = self.response_model
                self.model.structured_outputs = True
            else:
                # Otherwise we just want JSON
                self.model.response_format = {"type": "json_object"}
        else:
            self.model.response_format = None
            

If we don't set the flag in the model, then it will default to json mode, which MIGHT work, but the native support should be better. The naming we use is... confusing.