Bedrock timeout in 2m even after setting apiCallTimeout and apiCallAttemptTimeout to 5m/10m

[ShaunDeng]

Upcoming End-of-Support

• I acknowledge the upcoming end-of-support for AWS SDK for Java v1 was announced, and migration to AWS SDK for Java v2 is recommended.

Describe the bug

Timeout happened in 2 minutes

Expected Behavior

should not timeout before 10m passed

Current Behavior

software.amazon.awssdk.core.exception.SdkClientException: Unable to execute HTTP request: Read timed out
at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:111)
Suppressed: software.amazon.awssdk.core.exception.SdkClientException: Request attempt 1 failure: Unable to execute HTTP request: Read timed out
Suppressed: software.amazon.awssdk.core.exception.SdkClientException: Request attempt 2 failure: Unable to execute HTTP request: Read timed out
Suppressed: software.amazon.awssdk.core.exception.SdkClientException: Request attempt 3 failure: Unable to execute HTTP request: Read timed out
Suppressed: The stacktrace has been enhanced by Reactor, refer to additional information below:
Error has been observed at the following site(s):
*__checkpoint ⇢ Handler com.lenovo.tec.ai.codeg.gateway.controller.TextProcessController#chat(ChatCompletionsParam, String, String, String) [DispatcherHandler]
Original Stack Trace:
at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:111)
at software.amazon.awssdk.core.exception.SdkClientException.create(SdkClientException.java:47)
at software.amazon.awssdk.core.internal.http.pipeline.stages.utils.RetryableStageHelper2.setLastException(RetryableStageHelper2.java:226)
at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage2.execute(RetryableStage2.java:65)
at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage2.execute(RetryableStage2.java:36)
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:56)
at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:36)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.executeWithTimer(ApiCallTimeoutTrackingStage.java:80)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:60)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:42)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:50)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:32)
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:37)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:26)
at software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient$RequestExecutionBuilderImpl.execute(AmazonSyncHttpClient.java:224)
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.invoke(BaseSyncClientHandler.java:103)
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.doExecute(BaseSyncClientHandler.java:173)
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1(BaseSyncClientHandler.java:80)
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:182)
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:74)
at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45)
at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:53)
at software.amazon.awssdk.services.bedrockruntime.DefaultBedrockRuntimeClient.invokeModel(DefaultBedrockRuntimeClient.java:235)

Reproduction Steps

please see below code constructor(public AbstractBedrockApi(String modelId, AwsCredentialsProvider credentialsProvider, Region region, ObjectMapper objectMapper, Duration timeout)) and invocation method (internalInvocation) :

import com.fasterxml.jackson.annotation.JsonInclude;
import com.fasterxml.jackson.annotation.JsonProperty;
import com.fasterxml.jackson.annotation.JsonInclude.Include;
import com.fasterxml.jackson.core.JsonProcessingException;
import com.fasterxml.jackson.databind.ObjectMapper;
import java.io.UncheckedIOException;
import java.nio.charset.StandardCharsets;
import java.time.Duration;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.ai.model.ModelOptionsUtils;
import org.springframework.util.Assert;
import reactor.core.publisher.Flux;
import reactor.core.publisher.Sinks;
import reactor.core.publisher.Sinks.EmitFailureHandler;
import software.amazon.awssdk.auth.credentials.AwsCredentialsProvider;
import software.amazon.awssdk.auth.credentials.ProfileCredentialsProvider;
import software.amazon.awssdk.core.SdkBytes;
import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.services.bedrockruntime.BedrockRuntimeAsyncClient;
import software.amazon.awssdk.services.bedrockruntime.BedrockRuntimeAsyncClientBuilder;
import software.amazon.awssdk.services.bedrockruntime.BedrockRuntimeClient;
import software.amazon.awssdk.services.bedrockruntime.BedrockRuntimeClientBuilder;
import software.amazon.awssdk.services.bedrockruntime.model.InvokeModelRequest;
import software.amazon.awssdk.services.bedrockruntime.model.InvokeModelResponse;
import software.amazon.awssdk.services.bedrockruntime.model.InvokeModelWithResponseStreamRequest;
import software.amazon.awssdk.services.bedrockruntime.model.InvokeModelWithResponseStreamResponseHandler;
import software.amazon.awssdk.services.bedrockruntime.model.InvokeModelWithResponseStreamResponseHandler.Visitor;
/**

• @Date 2024/7/24
  */
  public class AbstractBedrockApi<I, O, SO> {
  private static final Logger logger = LoggerFactory.getLogger(org.springframework.ai.bedrock.api.AbstractBedrockApi.class);
  private final String modelId;
  private final ObjectMapper objectMapper;
  private final Region region;
  private final BedrockRuntimeClient client;
  private final BedrockRuntimeAsyncClient clientStreaming;

  public AbstractBedrockApi(String modelId, String region) {
  this(modelId, ProfileCredentialsProvider.builder().build(), (String)region, ModelOptionsUtils.OBJECT_MAPPER, Duration.ofMinutes(5L));
  }

  public AbstractBedrockApi(String modelId, String region, Duration timeout) {
  this(modelId, ProfileCredentialsProvider.builder().build(), (String)region, ModelOptionsUtils.OBJECT_MAPPER, timeout);
  }

  public AbstractBedrockApi(String modelId, AwsCredentialsProvider credentialsProvider, String region, ObjectMapper objectMapper) {
  this(modelId, credentialsProvider, region, objectMapper, Duration.ofMinutes(5L));
  }

  public AbstractBedrockApi(String modelId, AwsCredentialsProvider credentialsProvider, String region, ObjectMapper objectMapper, Duration timeout) {
  this(modelId, credentialsProvider, Region.of(region), objectMapper, timeout);
  }

  public AbstractBedrockApi(String modelId, AwsCredentialsProvider credentialsProvider, Region region, ObjectMapper objectMapper, Duration timeout) {
  Assert.hasText(modelId, "Model id must not be empty");
  Assert.notNull(credentialsProvider, "Credentials provider must not be null");
  Assert.notNull(region, "Region must not be empty");
  Assert.notNull(objectMapper, "Object mapper must not be null");
  Assert.notNull(timeout, "Timeout must not be null");
  this.modelId = modelId;
  this.objectMapper = objectMapper;
  this.region = region;
  this.client = (((BedrockRuntimeClient.builder().region(this.region)).credentialsProvider(credentialsProvider)).overrideConfiguration((c) -> {
  c.apiCallTimeout(timeout);
  c.apiCallAttemptTimeout(timeout);
  })).build();
  this.clientStreaming = (((BedrockRuntimeAsyncClient.builder().region(this.region)).credentialsProvider(credentialsProvider)).overrideConfiguration((c) -> {
  c.apiCallTimeout(timeout);
  c.apiCallAttemptTimeout(timeout);
  })).build();
  }

  public String getModelId() {
  return this.modelId;
  }

  public Region getRegion() {
  return this.region;
  }

  protected O embedding(I request) {
  throw new UnsupportedOperationException("Embedding is not supported for this model: " + this.modelId);
  }

  protected O chatCompletion(I request) {
  throw new UnsupportedOperationException("Chat completion is not supported for this model: " + this.modelId);
  }

  protected Flux chatCompletionStream(I request) {
  throw new UnsupportedOperationException("Streaming chat completion is not supported for this model: " + this.modelId);
  }

  protected O internalInvocation(I request, Class clazz) {
  SdkBytes body;
  try {
  body = SdkBytes.fromUtf8String(this.objectMapper.writeValueAsString(request));
  } catch (JsonProcessingException var9) {
  throw new IllegalArgumentException("Invalid JSON format for the input request: " + request, var9);
  }

   InvokeModelRequest invokeRequest = (InvokeModelRequest)InvokeModelRequest.builder().modelId(this.modelId).body(body).build();
   InvokeModelResponse response = this.client.invokeModel(invokeRequest);
   String responseBody = response.body().asString(StandardCharsets.UTF_8);
  
   try {
       return this.objectMapper.readValue(responseBody, clazz);
   } catch (UncheckedIOException | JsonProcessingException var8) {
       throw new IllegalArgumentException("Invalid JSON format for the response: " + responseBody, var8);
   }
  

  }

  protected Flux internalInvocationStream(I request, Class clazz) {
  Sinks.Many eventSink = Sinks.many().multicast().onBackpressureBuffer();

   SdkBytes body;
   try {
       body = SdkBytes.fromUtf8String(this.objectMapper.writeValueAsString(request));
   } catch (JsonProcessingException var8) {
       eventSink.tryEmitError(var8);
       return eventSink.asFlux();
   }
  
   InvokeModelWithResponseStreamRequest invokeRequest = (InvokeModelWithResponseStreamRequest)InvokeModelWithResponseStreamRequest.builder().modelId(this.modelId).body(body).build();
   InvokeModelWithResponseStreamResponseHandler.Visitor visitor = Visitor.builder().onChunk((chunk) -> {
       try {
           Logger var10000 = logger;
           SdkBytes var10001 = chunk.bytes();
           var10000.debug("Received chunk: " + var10001.asString(StandardCharsets.UTF_8));
           SO response = this.objectMapper.readValue(chunk.bytes().asByteArray(), clazz);
           eventSink.tryEmitNext(response);
       } catch (Exception var5) {
           logger.error("Failed to unmarshall", var5);
           eventSink.tryEmitError(var5);
       }
  
   }).onDefault((event) -> {
       logger.error("Unknown or unhandled event: " + event.toString());
       eventSink.tryEmitError(new Throwable("Unknown or unhandled event: " + event.toString()));
   }).build();
   InvokeModelWithResponseStreamResponseHandler responseHandler = ((InvokeModelWithResponseStreamResponseHandler.Builder)((InvokeModelWithResponseStreamResponseHandler.Builder)((InvokeModelWithResponseStreamResponseHandler.Builder)InvokeModelWithResponseStreamResponseHandler.builder().onComplete(() -> {
       for(Sinks.EmitResult emitResult = eventSink.tryEmitComplete(); !emitResult.isSuccess(); emitResult = eventSink.tryEmitComplete()) {
           System.out.println("Emitting complete:" + emitResult);
       }
  
       eventSink.emitComplete(EmitFailureHandler.busyLooping(Duration.ofSeconds(3L)));
       logger.debug("\nCompleted streaming response.");
   })).onError((error) -> {
       logger.error("\n\nError streaming response: " + error.getMessage());
       eventSink.tryEmitError(error);
   })).onEventStream((stream) -> {
       stream.subscribe((e) -> {
           e.accept(visitor);
       });
   })).build();
   this.clientStreaming.invokeModelWithResponseStream(invokeRequest, responseHandler);
   return eventSink.asFlux();
  

  }

  @JsonInclude(Include.NON_NULL)
  public static record AmazonBedrockInvocationMetrics(Long inputTokenCount, Long firstByteLatency, Long outputTokenCount, Long invocationLatency) {
  public AmazonBedrockInvocationMetrics(@JsonProperty("inputTokenCount") Long inputTokenCount, @JsonProperty("firstByteLatency") Long firstByteLatency, @JsonProperty("outputTokenCount") Long outputTokenCount, @JsonProperty("invocationLatency") Long invocationLatency) {
  this.inputTokenCount = inputTokenCount;
  this.firstByteLatency = firstByteLatency;
  this.outputTokenCount = outputTokenCount;
  this.invocationLatency = invocationLatency;
  }

   @JsonProperty("inputTokenCount")
   public Long inputTokenCount() {
       return this.inputTokenCount;
   }
  
   @JsonProperty("firstByteLatency")
   public Long firstByteLatency() {
       return this.firstByteLatency;
   }
  
   @JsonProperty("outputTokenCount")
   public Long outputTokenCount() {
       return this.outputTokenCount;
   }
  
   @JsonProperty("invocationLatency")
   public Long invocationLatency() {
       return this.invocationLatency;
   }
  

  }

Possible Solution

No response

Additional Information/Context

No response

AWS Java SDK version used

bedrockruntime 2.26.12

JDK version used

java17

Operating System and version

ubuntu22.04.1

[ShaunDeng]

Sorry, I should provide more information. I'm calling anthropic.claude-3-5-sonnet-20240620-v1:0 and have inputted a complicated task which the LLM normally takes 1-2 minutes to complete.

[ShaunDeng]

Below is The minimal code snippet to reproduce the bug.

`AWSBedrockLoginInfo awsLoginInfo = convertLoginInfo(item.getLoginInfo());
AIModelEnum aiModelEnum = AIModelEnum.of(item.getProcessorCode());
org.springframework.ai.bedrock.anthropic3.api.Anthropic3ChatBedrockApi.AnthropicChatRequest request = chatModel.createRequest(new Prompt(messages));
BedrockRuntimeClient client = (((BedrockRuntimeClient.builder().region(Region.of(awsLoginInfo.getRegion()))).credentialsProvider(StaticCredentialsProvider.create(AwsBasicCredentials.create(awsLoginInfo.getAccessKey(),awsLoginInfo.getSecretKey())))).overrideConfiguration((c) -> {
c.apiCallTimeout(timeout);
c.apiCallAttemptTimeout(timeout);
})).build();

        SdkBytes body;
        try {
            body = SdkBytes.fromUtf8String(ModelOptionsUtils.OBJECT_MAPPER.writeValueAsString(request));
        } catch (JsonProcessingException var9) {
            throw new IllegalArgumentException("Invalid JSON format for the input request: " + request, var9);
        }

        InvokeModelRequest invokeRequest = (InvokeModelRequest)InvokeModelRequest.builder().modelId(aiModelEnum.getRawCode()).body(body).build();
        client.invokeModel(invokeRequest);`

[debora-ito]

Moved to the Java SDK v2 repo.

[debora-ito]

So the read timeout is being triggered in this case, not the API call attempt timeout, otherwise you'd see a ApiCallAttemptTimeoutException.

Try increasing the NettyNioAsyncHttpClient readTimeout in the async client, and ApacheHttpClient socketTimeout in the sync client.

If that doesn't help, I recommend you enable the client-side metrics to have a sense of how long a response from Bedrock usually takes, and have a look at other metrics. Instructions can be found in our Developer Guide: https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/metrics.html

[ShaunDeng]

Thank you, debora. It seems like worked by setting NettyNioAsyncHttpClient and also for ApacheHttpClient, please see my below code snippet:

        AttributeMap timeoutConfig = AttributeMap.builder()
                .put(SdkHttpConfigurationOption.READ_TIMEOUT, timeout)
                .build();

        SdkHttpClient apacheHttpClient = ApacheHttpClient.builder().buildWithDefaults(timeoutConfig);

        SdkAsyncHttpClient nettyClient = NettyNioAsyncHttpClient.builder()
                .buildWithDefaults(timeoutConfig);

        this.client = (((BedrockRuntimeClient.builder().httpClient(apacheHttpClient).region(this.region)).credentialsProvider(credentialsProvider)).overrideConfiguration((c) -> {
            c.apiCallTimeout(timeout);
            c.apiCallAttemptTimeout(timeout);
        })).build();
        this.clientStreaming = (((BedrockRuntimeAsyncClient.builder().httpClient(nettyClient).region(this.region)).credentialsProvider(credentialsProvider)).overrideConfiguration((c) -> {
            c.apiCallTimeout(timeout);
            c.apiCallAttemptTimeout(timeout);
        })).build();

[github-actions]

This issue is now closed. Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.

[hrushig]

Check this out - in similar lines - https://repost.aws/articles/AR_A_6aqweQYOR5FCgvM2vdA/solving-read-timed-out-errors-and-high-latencies-in-amazon-bedrock-with-aws-java-sdk-client