Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enhancement: Support content array input message for anthropic messages #244

Closed
VisargD opened this issue Mar 5, 2024 · 1 comment · Fixed by #245
Closed

Enhancement: Support content array input message for anthropic messages #244

VisargD opened this issue Mar 5, 2024 · 1 comment · Fixed by #245
Assignees
Labels
enhancement New feature or request triage

Comments

@VisargD
Copy link
Collaborator

VisargD commented Mar 5, 2024

Anthropic now supports vision input with the release of claude-3 models. With this release, it has started accepting a different format for messages where content of a message can be an array and that array can have text and image inputs.

"messages": [
        {
            "role": "system",
            "content": "Your are a helpful assistant"
        },
        {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": "Whats in the image"
                },
                {
                    "type": "image_url",
                    "image_url": {
                        "url": "base64data"
                    }
                }
            ]
        }
    ]

The system message transformer written for anthropic and bedrock-anthropic chat completions only accepts the content as a string and not as an array. This should be changed now to support array as well so that the structure is uniform for all types of messages. Here is an example of system message which should also be supported in gateway:

"messages": [
        {
            "role": "system",
            "content": [
                {
                    "type": "text",
                    "text": "Whats in the image"
                }
            ]
        },
        {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": "Whats in the image"
                },
                {
                    "type": "image_url",
                    "image_url": {
                        "url": "base64data"
                    }
                }
            ]
        }
    ]

API DOCs reference for the new messages structure: https://docs.anthropic.com/claude/reference/messages_post

@flexchar
Copy link
Contributor

flexchar commented Mar 6, 2024

Thank you for this, Visarg! Do you have anything against migrating to the new 'messages API` from Claude?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request triage
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants