[Feature Request] Query Bedrock model maxTokens limit

[austinmw]

Describe the feature

Hi, I'd like to be able to query the maximum value for maxTokens for foundation models (especially ones using the Converse API)

Use Case

The documentation for the upper limit on maxTokens value for each model is not centrally located and is instead scattered across various documentation pages. For builders trying to support a wide range of models on Bedrock, being able to query this information is important.

For example, I might want to run something like:

model_id = 'sonnet 3.5'
max_tokens_limit = client.???(model_id)
client.converse(model_id, maxTokens = min(5000, max_tokens_limit)

Proposed Solution

Add boto3 capability for this

Other Information

No response

Acknowledgements

• I may be able to implement this feature request
• This feature might incur a breaking change

SDK version used

latest

Environment details (OS name and version, etc.)

latest

[tim-finnigan]

Thanks for reaching out. We'll have to reach out to the Bedrock team regarding this feature request, as they maintain the Converse API and token configurations. This documentation notes some limitations on token usage which may be helpful: https://docs.aws.amazon.com/bedrock/latest/userguide/quotas.html. There are also APIs for tracking service quotas, but I don't think specifically for what you're requesting.

Since APIs like Converse are used across AWS SDKs, I created aws/aws-sdk#780 in our cross-SDK repository to track this going forward. Please refer to that issue for updates.

[github-actions]

This issue is now closed. Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.