How can we use Bedrock Application Inference Profiles?

[mknet3]

Hi! is there a way to use Bedrock Application Inference Profiles ?

I checked Cross-Region Inference profile and it works fine with something like this:

      - model_name: anthropic-claude-3-7-sonnet
        litellm_params:
          model: bedrock/us.anthropic.claude-3-7-sonnet-20250219-v1:0
          aws_region_name: us-east-1

But, what about an application inference profile? I tested in many ways but I could not make it work. E.g.:

      - model_name: anthropic-claude-3-7-sonnet
        litellm_params:
          model: bedrock/arn:aws:bedrock:us-east-1:000000000000:application-inference-profile/0z0z0z0z0z0
          aws_region_name: us-east-1

also with this:

      - model_name: anthropic-claude-3-7-sonnet
        litellm_params:
          model: bedrock/anthropic/arn:aws:bedrock:us-east-1:000000000000:application-inference-profile/0z0z0z0z0z0
          model_id: bedrock/arn:aws:bedrock:us-east-1:000000000000:application-inference-profile/0z0z0z0z0z0
          aws_region_name: us-east-1

Application Inference Profiles are a strong requirement in AWS if you want to know usage / costs per application.

Any idea how to configure it? 🙏

[krrishdholakia]

maybe relevant to @mirodrr2's issue

[mknet3]

Just to link it here -> #8911
I think it is close related, I will look into the issue! Thanks!!

[mknet3]

I tested with the latest version (main-v1.63.11-nightly) and it works perfect. Thank you so much. This is how it works to me:

  - model_name: anthropic-claude-3-5-sonnet
    litellm_params:
      model: bedrock/anthropic.claude-3-5-sonnet-20240620-v1:0
      # You have to set the ARN application inference profile in the model_id parameter
      model_id: arn:aws:bedrock:eu-central-1:000000000000:application-inference-profile/a0a0a0a0a0a0
      aws_region_name: eu-central-1

[mknet3]

I get this error setting bedrock/converse/arn... while setting model_id works:

LiteLLM Proxy:ERROR: proxy_server.py:3063 - litellm.proxy.proxy_server.async_data_generator(): Exception occured - litellm.APIConnectionError: Header length of 1953527156 exceeded the maximum of 131072
Traceback (most recent call last):
  File "/usr/lib/python3.13/site-packages/litellm/litellm_core_utils/streaming_handler.py", line 1671, in __anext__
    async for chunk in self.completion_stream:
    ...<50 lines>...
        return processed_chunk
  File "/usr/lib/python3.13/site-packages/litellm/llms/bedrock/chat/invoke_handler.py", line 1471, in aiter_bytes
    for event in event_stream_buffer:
                 ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.13/site-packages/botocore/eventstream.py", line 544, in __next__
    return self.next()
           ~~~~~~~~~^^
  File "/usr/lib/python3.13/site-packages/botocore/eventstream.py", line 536, in next
    self._prelude = self._parse_prelude()
                    ~~~~~~~~~~~~~~~~~~~^^
  File "/usr/lib/python3.13/site-packages/botocore/eventstream.py", line 480, in _parse_prelude
    self._validate_prelude(prelude)
    ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
  File "/usr/lib/python3.13/site-packages/botocore/eventstream.py", line 471, in _validate_prelude
    raise InvalidHeadersLength(prelude.headers_length)
botocore.eventstream.InvalidHeadersLength: Header length of 1953527156 exceeded the maximum of 131072

[kderholtvisma]

any solution to that header issue?

[krrishdholakia]

I wasn't able to debug the issue. I could see it being passed in correctly to the aws sdk. If anyone has a fix - a PR here is welcome!

[tmwilder]

As a +1 - a bunch of engineers I work with are running into this one on Sonnet/Opus 4.0 and 4.1 and it makes lightllm unusable for those models.

[tmwilder]

It's not fully deterministic - it happens mid-session and retries sometimes power through, frequency appears to go up with longer sessions. I think it's happening when a specific api call is longer like after getting the results of a CLI call or test suite execution.

This one is breaking enough that it's a reason for people to find ways to disintermediate lightllm.