Exception occurs one hour after execution: (Connection pool shut down) #4386

dolphin235 · 2023-09-01T09:20:12Z

Describe the bug

This error has been occurring since upgrading from version 2.16.1 to 2.19.14.
It doesn't occur in all cases, but only when using specific credentials. It works well when receiving credentials through IMDS, but only occurs when using the file path of AWS_WEB_IDENTITY_TOKEN_FILE as an environment variable.
Under the two conditions mentioned, it works well for one hour, but once one hour passes, the following error occurs.

It's difficult to identify the exact cause. If there are any suspected issues, please let me know.
Could I have made a mistake in using the SDK?

java.lang.IllegalStateException: Connection pool shut down
        at org.apache.http.util.Asserts.check(Asserts.java:34)
        at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.requestConnection(PoolingHttpClientConnectionManager.java:269)
        at software.amazon.awssdk.http.apache.internal.conn.ClientConnectionManagerFactory$DelegatingHttpClientConnectionManager.requestConnection(ClientConnectionManagerFactory.java:75)
        at software.amazon.awssdk.http.apache.internal.conn.ClientConnectionManagerFactory$InstrumentedHttpClientConnectionManager.requestConnection(ClientConnectionManagerFactory.java:57)
        at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:176)
        at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
        at software.amazon.awssdk.http.apache.internal.impl.ApacheSdkHttpClient.execute(ApacheSdkHttpClient.java:72)
        at software.amazon.awssdk.http.apache.ApacheHttpClient.execute(ApacheHttpClient.java:254)
        at software.amazon.awssdk.http.apache.ApacheHttpClient.access$500(ApacheHttpClient.java:104)
        at software.amazon.awssdk.http.apache.ApacheHttpClient$1.call(ApacheHttpClient.java:231)
        at software.amazon.awssdk.http.apache.ApacheHttpClient$1.call(ApacheHttpClient.java:228)
        at software.amazon.awssdk.core.internal.util.MetricUtils.measureDurationUnsafe(MetricUtils.java:63)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.executeHttpRequest(MakeHttpRequestStage.java:77)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.execute(MakeHttpRequestStage.java:56)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.execute(MakeHttpRequestStage.java:39)
        at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
        at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
        at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
        at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:73)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:42)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:78)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:40)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:50)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:36)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:81)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:36)
        at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
        at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:56)
        at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:36)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.executeWithTimer(ApiCallTimeoutTrackingStage.java:80)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:60)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:42)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:48)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:31)
        at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
        at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:37)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:26)
        at software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient$RequestExecutionBuilderImpl.execute(AmazonSyncHttpClient.java:193)
        at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.invoke(BaseSyncClientHandler.java:103)
        at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.doExecute(BaseSyncClientHandler.java:171)
        at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1(BaseSyncClientHandler.java:82)
        at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:179)
        at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:76)
        at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45)
        at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:56)
        at software.amazon.awssdk.services.sts.DefaultStsClient.assumeRoleWithWebIdentity(DefaultStsClient.java:760)
        at software.amazon.awssdk.services.sts.auth.StsAssumeRoleWithWebIdentityCredentialsProvider.getUpdatedCredentials(StsAssumeRoleWithWebIdentityCredentialsProvider.java:73)
        at software.amazon.awssdk.services.sts.auth.StsCredentialsProvider.updateSessionCredentials(StsCredentialsProvider.java:88)
        at software.amazon.awssdk.utils.cache.CachedSupplier.lambda$jitteredPrefetchValueSupplier$3(CachedSupplier.java:283)
        at software.amazon.awssdk.utils.cache.CachedSupplier$PrefetchStrategy.fetch(CachedSupplier.java:419)
        at software.amazon.awssdk.utils.cache.CachedSupplier.refreshCache(CachedSupplier.java:198)
        at software.amazon.awssdk.utils.cache.CachedSupplier.get(CachedSupplier.java:127)
        at software.amazon.awssdk.services.sts.auth.StsCredentialsProvider.resolveCredentials(StsCredentialsProvider.java:99)
        at software.amazon.awssdk.services.sts.auth.StsAssumeRoleWithWebIdentityCredentialsProvider.resolveCredentials(StsAssumeRoleWithWebIdentityCredentialsProvider.java:44)
        at software.amazon.awssdk.services.sts.internal.StsWebIdentityCredentialsProviderFactory$StsWebIdentityCredentialsProvider.resolveCredentials(StsWebIdentityCredentialsProviderFactory.java:93)
        at software.amazon.awssdk.auth.credentials.WebIdentityTokenFileCredentialsProvider.resolveCredentials(WebIdentityTokenFileCredentialsProvider.java:113)
        at software.amazon.awssdk.auth.credentials.AwsCredentialsProviderChain.resolveCredentials(AwsCredentialsProviderChain.java:90)
        at software.amazon.awssdk.auth.credentials.internal.LazyAwsCredentialsProvider.resolveCredentials(LazyAwsCredentialsProvider.java:45)
        at software.amazon.awssdk.auth.credentials.DefaultCredentialsProvider.resolveCredentials(DefaultCredentialsProvider.java:123)

Expected Behavior

I expect it to continue functioning properly even as time passes.

Current Behavior

An error occurs after one hour has passed.

Reproduction Steps

When running this code with 2.19.14 version, an error occurs after one hour has passed.

import java.net.URI;

import software.amazon.awssdk.auth.credentials.AwsCredentials;
import software.amazon.awssdk.auth.credentials.DefaultCredentialsProvider;
import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.auth.signer.Aws4Signer;
import software.amazon.awssdk.auth.signer.params.Aws4SignerParams;
import software.amazon.awssdk.http.SdkHttpFullRequest;
import software.amazon.awssdk.http.SdkHttpMethod;


            for (int i=0;i<150;i++){
                try(DefaultCredentialsProvider defaultCredentialsProvider = DefaultCredentialsProvider.create()) {
                    AwsCredentials credentials = defaultCredentialsProvider.resolveCredentials();

                    Region region = Region.AP_NORTHEAST_2;
                    Aws4Signer aws4Signer = Aws4Signer.create();
                    String sts_url = String.format("sts.%s.amazonaws.com", region);
                    SdkHttpFullRequest request = SdkHttpFullRequest.builder()
                        .method(SdkHttpMethod.POST)
                        .appendHeader("Host", sts_url)
                        .uri(URI.create(String.format("https://%s/?Action=GetCallerIdentity&Version=2011-06-15", sts_url)))
                        .build();
                    SdkHttpFullRequest signedRequest = aws4Signer.sign(request, Aws4SignerParams.builder().awsCredentials(credentials).signingName("sts").signingRegion(region).build());

                    Thread.sleep(1000 * 60);
                }
            }

Possible Solution

No response

Additional Information/Context

No response

AWS Java SDK version used

2.19.14

JDK version used

openjdk 11.0.20.1 2023-08-24

Operating System and version

Ubuntu 22.04.2 LTS

The text was updated successfully, but these errors were encountered:

debora-ito · 2023-09-09T00:39:15Z

WebIdentityTokenFileCredentialsProvider became AutoCloseable in SDK version 2.17.283 via #3440.

So, in your code example, defaultCredentialsProvider will close all the providers in the chain after the try block ends.

dolphin235 · 2023-09-11T02:12:42Z

Thank you for the answer.

github-actions · 2023-09-11T17:40:21Z

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

q-nathangrand · 2023-11-24T17:24:10Z

Just to save someone else scratching their head for a while - DefaultCredentialsProvider.create() doesn't actually create a new instance each iteration of the for loop, so once it's closed and the end of the first try the (single, 'global') credentials provider is then closed. My understanding is this goes unnoticed until it attempts to refresh a token after 1 hour, at which point the error occurs.

steveloughran · 2023-11-25T13:31:43Z

Fun one. It means that on any process > 1h duration, if the first instance of the credential provider is closed, all other uses fail. Just had to review our code to make sure we don't do this. All good in s3afs as (a) we don't use this chain and (b) our list is ref counted (org.apache.hadoop.fs.s3a.AWSCredentialProviderList) and only closes itself when the ref count is 0. aws sdk should consider this

The workaround to do here is wrap the default chain in your own credential provider implementation, one which doesn't implement close().

q-nathangrand · 2023-11-26T12:47:40Z

It's particularly 'fun' as the javadocs say:

/**
* Create an instance of the {@link DefaultCredentialsProvider} using the default configuration. Configuration can be
* specified by creating an instance using the {@link #builder()}.
*/
public static DefaultCredentialsProvider create()

Which isn't really true. Could be better named (maybe getOrCreate like Spark do it on a SparkSession) or actually create a new instance each time it's called.

steveloughran · 2023-11-28T19:09:11Z

type 3 fun: https://www.tetongravity.com/story/adventure/the-three-and-a-half-types-of-fun-explained

nsvijay04b1 · 2023-11-29T09:36:47Z

Any working code sample how to use credentials or refresh so that it still works after 1 hr , please ?

I see similar issues are closed without a working solution suggested but answering this is expected behavior.

Error:- java.lang.IllegalStateException: Connection pool shut down

Code:-

try(CloudWatchClient cw = CloudWatchClient.builder().region(region).credentialsProvider(DefaultCredentialsProvider.create()).build()){

// code

}

q-nathangrand · 2023-11-29T11:36:28Z

Switching DefaultCredentialsProvider.create() for DefaultCredentialsProvider.builder().build() in your code would be the equivalent to how I solved it. That way, you build a new CredentialsProvider per CloudWatchClient, which allows you to safely close your CloudWatchClient

nsvijay04b1 · 2023-11-30T08:29:19Z

Switching DefaultCredentialsProvider.create() for DefaultCredentialsProvider.builder().build() in your code would be the equivalent to how I solved it. That way, you build a new CredentialsProvider per CloudWatchClient, which allows you to safely close your CloudWatchClient

Thanks @q-nathangrand DefaultCredentialsProvider.builder().build() - Fixed it.

dolphin235 added bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels Sep 1, 2023

debora-ito added closing-soon This issue will close in 4 days unless further comments are made. and removed needs-triage This issue or PR still needs to be triaged. labels Sep 9, 2023

debora-ito self-assigned this Sep 9, 2023

debora-ito added guidance Question that needs advice or information. and removed bug This issue is a bug. labels Sep 9, 2023

github-actions bot removed the closing-soon This issue will close in 4 days unless further comments are made. label Sep 11, 2023

debora-ito closed this as not planned Won't fix, can't repro, duplicate, stale Sep 11, 2023

piby180 mentioned this issue Oct 9, 2023

PinotS3 : Connection Pool Shutdown after one hour on EKS on release-1.0.0 apache/pinot#11761

Closed

zoewangg mentioned this issue Mar 8, 2024

Update documentation for DefaultCredentialsProvider.create #5009

Merged

chenlin2001 mentioned this issue Mar 18, 2024

Mysterious "Connection Pool Closed" Error #4221

Closed

pwannenmacher mentioned this issue Apr 18, 2024

AWS STS: IllegalStateException "Connection pool shut down" Frachtwerk/essencium-backend-file-storage#195

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Exception occurs one hour after execution: (Connection pool shut down) #4386

Exception occurs one hour after execution: (Connection pool shut down) #4386

dolphin235 commented Sep 1, 2023 •

edited

Loading

debora-ito commented Sep 9, 2023

dolphin235 commented Sep 11, 2023

github-actions bot commented Sep 11, 2023

q-nathangrand commented Nov 24, 2023

steveloughran commented Nov 25, 2023 •

edited

Loading

q-nathangrand commented Nov 26, 2023 •

edited

Loading

steveloughran commented Nov 28, 2023

nsvijay04b1 commented Nov 29, 2023 •

edited

Loading

q-nathangrand commented Nov 29, 2023

nsvijay04b1 commented Nov 30, 2023

Exception occurs one hour after execution: (Connection pool shut down) #4386

Exception occurs one hour after execution: (Connection pool shut down) #4386

Comments

dolphin235 commented Sep 1, 2023 • edited Loading

Describe the bug

Expected Behavior

Current Behavior

Reproduction Steps

Possible Solution

Additional Information/Context

AWS Java SDK version used

JDK version used

Operating System and version

debora-ito commented Sep 9, 2023

dolphin235 commented Sep 11, 2023

github-actions bot commented Sep 11, 2023

⚠️COMMENT VISIBILITY WARNING⚠️

q-nathangrand commented Nov 24, 2023

steveloughran commented Nov 25, 2023 • edited Loading

q-nathangrand commented Nov 26, 2023 • edited Loading

steveloughran commented Nov 28, 2023

nsvijay04b1 commented Nov 29, 2023 • edited Loading

q-nathangrand commented Nov 29, 2023

nsvijay04b1 commented Nov 30, 2023

dolphin235 commented Sep 1, 2023 •

edited

Loading

steveloughran commented Nov 25, 2023 •

edited

Loading

q-nathangrand commented Nov 26, 2023 •

edited

Loading

nsvijay04b1 commented Nov 29, 2023 •

edited

Loading