Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cannot handle properly when bedrock response with empty string #36

Open
1 task done
Shellmode opened this issue Aug 9, 2024 · 2 comments
Open
1 task done

Cannot handle properly when bedrock response with empty string #36

Shellmode opened this issue Aug 9, 2024 · 2 comments
Labels
bug Something isn't working

Comments

@Shellmode
Copy link

Describe the bug
当 bedrock 返回空字符串的时候,在 https://github.com/aws-samples/bedrock-access-gateway/blob/main/src/api/models/bedrock.py#L456 处会报错,trace 如下

image

Please complete the following information:

  • Which API you used: [e.g. /chat/completions]
  • [] Which model you used: [e.g. anthropic.claude-3-sonnet-20240229-v1:0]

To Reproduce
Steps to reproduce the behavior. If possible, please share an example request.

curl http://Bedroc-Proxy-abcdefg-123123.us-west-2.elb.amazonaws.com/api/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer bedrock" \
  -d '{
    "model": "anthropic.claude-3-5-sonnet-20240620-v1:0",
    "messages": [
      {"role": "user", "content": "Given a [text fragment], extract the person names from it. Note that: 1. List the person names directly, separated by commas, like this: Mary, Mike, Lucy. 2. If no characters are recognized in the [text fragment], return an empty string directly, without adding any explanations or comments. 3. Do not output any additional text or explanations. Text fragment: \"Darkness.\"He suddenly woke up, and that was his first thought.\" Extracting result:"}
    ]
  }'

Expected behavior
Response with empty string

Screenshots
image

Additional context
Add any other context about the problem here.

@Shellmode Shellmode added the bug Something isn't working label Aug 9, 2024
@daixba
Copy link
Contributor

daixba commented Aug 9, 2024

我测试了下, 是因为 bedrock 返回content 为空, 如下所示:
'message': {'role': 'assistant', 'content': []}}, 'stopReason': 'end_turn',

针对这种情况就返回为空:

{"id":"chatcmpl-a3aa1356","created":1723196261,"model":"anthropic.claude-3-5-sonnet-20240620-v1:0","system_fingerprint":"fp","choices":[{"index":0,"finish_reason":"stop","logprobs":null,"message":{"role":"assistant"}}],"object":"chat.completion","usage":{"prompt_tokens":112,"completion_tokens":4,"total_tokens":116}}

fix 代码见: 2950c15

@daixba
Copy link
Contributor

daixba commented Aug 9, 2024

参考 https://github.com/aws-samples/bedrock-access-gateway/blob/main/README_CN.md#%E5%A6%82%E4%BD%95%E5%8D%87%E7%BA%A7 进行更新

要使用最新功能,您无需重新部署CloudFormation堆栈。您只需拉取最新的镜像即可。

具体操作方式取决于您部署的版本:

Lambda版本: 进入AWS Lambda控制台,找到Lambda 函数,然后找到并单击部署新映像按钮,然后单击保存。
Fargate版本: 进入ECS控制台,单击ECS集群,转到任务选项卡,选择正在运行的唯一任务,然后点击停止所选菜单, ECS会自动启动新任务并且使用最新镜像。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants