Mysterious "Connection Pool Closed" Error

[Khongchai]

Describe the bug

Calling putObject with the s3AsyncClient from software.amazon.awssdk.services.s3.S3AsyncClient while using WebIdentityTokenFileCredentialsProvider as the credential provider will cause the "connection pool closed" error from AwsSyncClientHandler the STS library uses.

We were using AWS SDK version 2.20.19. Upgrading to newer ones didn't solve our problem, but when we downgraded to 2.16.18 the problem disappeared.

When this error happens, all subsequent requests to the endpoint that uses this library will throw the same error, the only solution is to restart the microservice. I will provide the stack trace here, this stack shows the assertion error at the HTTP client level, but we don't know who closed it (still trying to reproduce it locally so we can use the debugger to help us).

Expected Behavior

The error should not be thrown. We didn't call any close on any HTTP client or thread pools.

Current Behavior

Once we deploy our microservice, it will function normally, then fail within an hour -- likely after having handled a number of requests from our test team.

Reproduction Steps

Initialization

final var httpClient = SdkAsyncHttpClient httpClient = NettyNioAsyncHttpClient.builder()
                .connectionTimeout(Duration.ofMillis(60000)
                .writeTimeout(Duration.ofMillis(60000)
                .readTimeout(Duration.ofMillis(60000)
                .maxConcurrency(8)
                .maxPendingConnectionAcquires(10000)
                .build();
final var credentialsProvider =  WebIdentityTokenFileCredentialsProvider.builder()
                    .roleArn(assumeRoleARN)
                    .webIdentityTokenFile(Path.of(tokenPath))
                    .build();
S3AsyncClientBuilder s3AsyncClientBuilder = S3AsyncClient.builder()
                .httpClient(httpClient)
                .region("ap-southeast-1")
                .credentialsProvider(credentialsProvider)
                .serviceConfiguration(serviceConfiguration)  // S3Configuration instance
                .build();

this. asyncClient = s3AsyncClientBuilder.build(); 

Calling putObject

asyncClient 
                .putObject(
                        PutObjectRequest.builder()
                                .bucket(...)
                                .contentLength(...)
                                .contentType(...)
                                .metadata(...)
                                .key(...)
                                .build(),
                        fileByte
                );

Calling putObject with this setup code a few times...or sporadically for an hour and the error appeared.

Possible Solution

No response

Additional Information/Context

No response

AWS Java SDK version used

2.20.19

JDK version used

jdk-11

Operating System and version

"linux", version: "3.10.0-1160.6.1.el7.x86_64", arch: "amd64", family: "unix"

[debora-ito]

"Connection Pool Closed" indicates the connection pool was closed prematurely.

The SDK does not close pools that are active. If you didn't explicitly closed the client, look for processes external to the SDK that can be closing the STS client, sometimes indirectly. Also, investigate the logs for any OutOfMemory errors. Sometimes the underlying HTTP client closes the connection pool if there's an error.

Can you provide the stacktrace that shows the "Connection Pool Closed" message?
It does not show in the printscreen of the stacktrace.

Moving this to the aws-sdk-java-v2 repo.

[github-actions]

It looks like this issue has not been active for more than five days. In the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please add a comment to prevent automatic closure, or if the issue is already closed please feel free to reopen it.

[pewtpong]

Hello, I am here to continue Khongchai's work. Our behavior is similar to the issue described in this link: [BUG: S3 Connection pool shut down for S3Client](aws/aws-sdk-java#1282) Currently, I am in the process of reproducing the problem, and I will promptly post the results of the reproduction once it is completed.

[debora-ito]

@pewtpong noted.

[github-actions]

It looks like this issue has not been active for more than five days. In the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please add a comment to prevent automatic closure, or if the issue is already closed please feel free to reopen it.

[chenlin2001]

I can see the same issue. I'm using Micronaut aws sdk which creates the credentials this way:
` */
@factory
@BootstrapContextCompatible
public class CredentialsAndRegionFactory {

/**
 * @param environment The {@link Environment}
 * @return An {@link AwsCredentialsProviderChain} that attempts to read the values from the Micronaut environment
 * first, then delegates to {@link DefaultCredentialsProvider}.
 */
@Bean(preDestroy = "close")
@Singleton
public AwsCredentialsProviderChain awsCredentialsProvider(Environment environment) {
    return AwsCredentialsProviderChain.of(
            EnvironmentAwsCredentialsProvider.create(environment),
            DefaultCredentialsProvider.create()
    );
}

`
When using STS to assume role with services running in Kuberenets, things works ok in the beginning, but after an hour I see the same issues of connection pool. Seems to be the same error as in this ticket and aws/aws-sdk-java#1282 and #4386
Any solution? This ticket seems to keep getting closed without solutions