StackOverflowError on Alpine when using DefaultS3CrtAsyncClient #4935

ffried · 2024-02-15T06:52:16Z

Describe the bug

Using the DefaultS3CrtAsyncClient as produced by S3AsyncClient.crtBuilder().build() fails with a StackOverflowError on Alpine when running a ListObjectsRequest.

Using S3AsyncClient.builder().build() (no CRT) succeeds.

Expected Behavior

The CRT client works just as fine as the normal client (if not better 😉).

Current Behavior

A StackOverflowError is raised.

Reproduction Steps

Reproduction files:

Main.java

package de.sersoft;

import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.services.s3.S3AsyncClient;

public class Main {
    public static void main(String[] args) {
        var bucketName = args[0];
        try (var s3 = S3AsyncClient.crtBuilder().region(Region.US_EAST_1).build()) {
            System.out.println("Listing objects in bucket " + bucketName);
            var objects = s3.listObjects(builder -> builder.bucket(bucketName)).join().contents();
            System.out.println("Objects in bucket " + bucketName + ":");
            for (var myValue : objects) {
                System.out.println("-" + myValue.key());
            }
        }
    }
}

pom.xml

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>de.sersoft</groupId>
    <artifactId>AWSAlpine</artifactId>
    <version>1.0.0-SNAPSHOT</version>
    <packaging>jar</packaging>

    <properties>
        <maven.compiler.source>21</maven.compiler.source>
        <maven.compiler.target>21</maven.compiler.target>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    </properties>

    <dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>software.amazon.awssdk</groupId>
                <artifactId>bom</artifactId>
                <version>2.22.2</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
        </dependencies>
    </dependencyManagement>

    <dependencies>
        <dependency>
            <groupId>software.amazon.awssdk</groupId>
            <artifactId>s3</artifactId>
        </dependency>
        <dependency>
            <groupId>software.amazon.awssdk.crt</groupId>
            <artifactId>aws-crt</artifactId>
            <version>0.29.2</version>
        </dependency>
    </dependencies>

    <build>
        <finalName>${project.artifactId}</finalName>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-assembly-plugin</artifactId>
                <version>3.6.0</version>
                <executions>
                    <execution>
                        <phase>package</phase>
                        <goals><goal>single</goal></goals>
                        <configuration>
                            <archive>
                                <manifest><mainClass>de.sersoft.Main</mainClass></manifest>
                            </archive>
                            <descriptorRefs>
                                <descriptorRef>jar-with-dependencies</descriptorRef>
                            </descriptorRefs>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>
</project>

Dockerfile

FROM --platform=$BUILDPLATFORM maven:3-eclipse-temurin-21-alpine as buildnode

WORKDIR /aws-alpine

COPY . .

RUN --mount=type=cache,target=/root/.m2/repository mvn clean package -DskipTests


FROM eclipse-temurin:21-jdk-alpine
WORKDIR /aws-alpine

COPY --from=buildnode /aws-alpine/target/AWSAlpine-jar-with-dependencies.jar ./

ENTRYPOINT ["java", "-jar", "/aws-alpine/AWSAlpine-jar-with-dependencies.jar" ]

Then building (with buildx) and running the container, passing a bucket name as command will result in a StackOverflowError.

Stack Trace (from stdout)

2023-12-21T06:50:27.806548589Z Exception in thread "main" java.util.concurrent.CompletionException: software.amazon.awssdk.core.exception.SdkClientException: Unable to execute HTTP request: null
2023-12-21T06:50:27.806597422Z 	at software.amazon.awssdk.utils.CompletableFutureUtils.errorAsCompletionException(CompletableFutureUtils.java:65)
2023-12-21T06:50:27.806692589Z 	at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncExecutionFailureExceptionReportingStage.lambda$execute$0(AsyncExecutionFailureExceptionReportingStage.java:51)
2023-12-21T06:50:27.806713506Z 	at java.base/java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:934)
2023-12-21T06:50:27.806720256Z 	at java.base/java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:911)
2023-12-21T06:50:27.806724006Z 	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
2023-12-21T06:50:27.806728672Z 	at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2194)
2023-12-21T06:50:27.806730464Z 	at software.amazon.awssdk.utils.CompletableFutureUtils.lambda$forwardExceptionTo$0(CompletableFutureUtils.java:79)
2023-12-21T06:50:27.806732089Z 	at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
2023-12-21T06:50:27.806772214Z 	at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
2023-12-21T06:50:27.806776714Z 	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
2023-12-21T06:50:27.806778464Z 	at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2194)
2023-12-21T06:50:27.806780172Z 	at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.maybeAttemptExecute(AsyncRetryableStage.java:103)
2023-12-21T06:50:27.806781922Z 	at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.maybeRetryExecute(AsyncRetryableStage.java:184)
2023-12-21T06:50:27.806783672Z 	at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.lambda$attemptExecute$1(AsyncRetryableStage.java:159)
2023-12-21T06:50:27.806785381Z 	at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
2023-12-21T06:50:27.806787006Z 	at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
2023-12-21T06:50:27.806806672Z 	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
2023-12-21T06:50:27.806820339Z 	at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2194)
2023-12-21T06:50:27.806826006Z 	at software.amazon.awssdk.utils.CompletableFutureUtils.lambda$forwardExceptionTo$0(CompletableFutureUtils.java:79)
2023-12-21T06:50:27.806828006Z 	at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
2023-12-21T06:50:27.806829797Z 	at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
2023-12-21T06:50:27.806831547Z 	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
2023-12-21T06:50:27.806833172Z 	at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2194)
2023-12-21T06:50:27.806834797Z 	at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.lambda$execute$0(MakeAsyncHttpRequestStage.java:108)
2023-12-21T06:50:27.806836506Z 	at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
2023-12-21T06:50:27.806838131Z 	at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
2023-12-21T06:50:27.806840422Z 	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
2023-12-21T06:50:27.807046089Z 	at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2194)
2023-12-21T06:50:27.807053172Z 	at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.completeResponseFuture(MakeAsyncHttpRequestStage.java:255)
2023-12-21T06:50:27.807056589Z 	at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.lambda$executeHttpRequest$3(MakeAsyncHttpRequestStage.java:167)
2023-12-21T06:50:27.807058547Z 	at java.base/java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:934)
2023-12-21T06:50:27.807060256Z 	at java.base/java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:911)
2023-12-21T06:50:27.807061839Z 	at java.base/java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:482)
2023-12-21T06:50:27.807063422Z 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
2023-12-21T06:50:27.807072839Z 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
2023-12-21T06:50:27.807074464Z 	at java.base/java.lang.Thread.run(Thread.java:1583)
2023-12-21T06:50:27.807076047Z Caused by: software.amazon.awssdk.core.exception.SdkClientException: Unable to execute HTTP request: null
2023-12-21T06:50:27.807077714Z 	at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:111)
2023-12-21T06:50:27.807080131Z 	at software.amazon.awssdk.core.exception.SdkClientException.create(SdkClientException.java:47)
2023-12-21T06:50:27.807094964Z 	at software.amazon.awssdk.core.internal.http.pipeline.stages.utils.RetryableStageHelper.setLastException(RetryableStageHelper.java:223)
2023-12-21T06:50:27.807097506Z 	at software.amazon.awssdk.core.internal.http.pipeline.stages.utils.RetryableStageHelper.setLastException(RetryableStageHelper.java:218)
2023-12-21T06:50:27.807099172Z 	at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.maybeRetryExecute(AsyncRetryableStage.java:182)
2023-12-21T06:50:27.807100839Z 	... 23 more
2023-12-21T06:50:27.807627297Z Caused by: java.lang.StackOverflowError
2023-12-21T06:50:27.807646922Z 	at java.base/java.lang.ClassLoader.defineClass1(Native Method)
2023-12-21T06:50:27.807649256Z 	at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1027)
2023-12-21T06:50:27.807650922Z 	at java.base/java.security.SecureClassLoader.defineClass(SecureClassLoader.java:150)
2023-12-21T06:50:27.807652547Z 	at java.base/jdk.internal.loader.BuiltinClassLoader.defineClass(BuiltinClassLoader.java:862)
2023-12-21T06:50:27.807654131Z 	at java.base/jdk.internal.loader.BuiltinClassLoader.findClassOnClassPathOrNull(BuiltinClassLoader.java:760)
2023-12-21T06:50:27.807655756Z 	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(BuiltinClassLoader.java:681)
2023-12-21T06:50:27.807657297Z 	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:639)
2023-12-21T06:50:27.807658881Z 	at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
2023-12-21T06:50:27.807660464Z 	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:526)
2023-12-21T06:50:27.807662006Z 	at software.amazon.awssdk.core.internal.handler.BaseAsyncClientHandler.lambda$new$0(BaseAsyncClientHandler.java:66)
2023-12-21T06:50:27.807663631Z 	at software.amazon.awssdk.core.internal.http.async.AsyncResponseHandler.lambda$prepare$0(AsyncResponseHandler.java:92)
2023-12-21T06:50:27.807761047Z 	at java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1150)
2023-12-21T06:50:27.807773339Z 	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
2023-12-21T06:50:27.807775422Z 	at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2179)
2023-12-21T06:50:27.807777172Z 	at software.amazon.awssdk.core.internal.http.async.AsyncResponseHandler$BaosSubscriber.onComplete(AsyncResponseHandler.java:135)
2023-12-21T06:50:27.807778839Z 	at software.amazon.awssdk.core.internal.metrics.BytesReadTrackingPublisher$BytesReadTracker.onComplete(BytesReadTrackingPublisher.java:74)
2023-12-21T06:50:27.807780506Z 	at software.amazon.awssdk.utils.async.SimplePublisher.doProcessQueue(SimplePublisher.java:275)
2023-12-21T06:50:27.807782089Z 	at software.amazon.awssdk.utils.async.SimplePublisher.processEventQueue(SimplePublisher.java:224)
2023-12-21T06:50:27.807789631Z 	at software.amazon.awssdk.utils.async.SimplePublisher.complete(SimplePublisher.java:157)
2023-12-21T06:50:27.807791297Z 	at software.amazon.awssdk.services.s3.internal.crt.S3CrtResponseHandlerAdapter.onSuccessfulResponseComplete(S3CrtResponseHandlerAdapter.java:106)
2023-12-21T06:50:27.807792964Z 	at software.amazon.awssdk.services.s3.internal.crt.S3CrtResponseHandlerAdapter.onFinished(S3CrtResponseHandlerAdapter.java:101)
2023-12-21T06:50:27.807794589Z 	at software.amazon.awssdk.crt.s3.S3MetaRequestResponseHandlerNativeAdapter.onFinished(S3MetaRequestResponseHandlerNativeAdapter.java:25)

Possible Solution

No response

Additional Information/Context

Using a non-alpine image (removing the -alpine suffixes in the build layer and run layer base image) does not result in the same exception.

This issue was originally reported in the aws-crt-java repository: awslabs/aws-crt-java#737
However, I was asked to report it here instead.

AWS Java SDK version used

2.22.2

JDK version used

21.0.1

Operating System and version

Alpine Linux v3.18

The text was updated successfully, but these errors were encountered:

tedahn · 2024-02-15T07:20:40Z

So this issue occurred for me as well…
we are using some modified version of: amazoncorretto:17 jdk-alpine 3.17

I’m just here to mention we switched the image to amazoncorretto 17 jdk ubuntu 20.04
And it worked 🥹

(But of course you can also use non-crt3 as an alternative…)

debora-ito · 2024-02-19T19:30:07Z

@ffried thank you for reaching out - and bonus points for using collapsible sections in the repro steps. We'll investigate.

JeHuiPark · 2024-03-14T06:51:36Z

I am also encountering this issue.

sdk version: 2.25.7
runtime image: eclipse-temurin:17-jre-alpine

gweekleyprovidertrust · 2024-03-14T21:45:17Z

I am encountering this issue with:

software.amazon.awssdk 2.25.8
software.amazon.awssdk.crt:aws-crt 0.29.11
runtime image eclipse-temurin:21-alpine

agebhar1 · 2024-03-22T17:27:39Z

This problem also occurs when the official Alpine-based Apache Kafka container is used in Eventbridge tests. A simplified version of the failing execution:

try (var ebClient = EventBridgeAsyncClient.builder()
        .region(US_EAST_1)
        .endpointOverride(ENDPOINT)
        .httpClientBuilder(AwsCrtAsyncHttpClient.builder())
        .overrideConfiguration(clientConfig)
        .credentialsProvider(DefaultCredentialsProvider.create())
        .build()
) {
    var entry = PutEventsRequestEntry.builder()
            .eventBusName(EVENTBUS_ARN)
            .source("kafka-connect.eventbridge-e2e-connector")
            .detailType("kafka-connect-eventbridge-e2e")
            .detail("{\"topic\":\"eventbridge-e2e\",\"partition\":0,\"offset\":0,\"timestamp\":1688464495808,\"timestampType\":\"CreateTime\",\"headers\":[],\"key\":null,\"value\":{\"hello\":\"world\"}}")
            .build();
    var request = PutEventsRequest.builder().entries(entry).build();
    logger.info("{}", request);

    try {
        var response = ebClient.putEvents(request).get(SDK_TIMEOUT, MILLISECONDS);
        logger.info("{}", response);
    } catch (InterruptedException | ExecutionException | TimeoutException e) {
        logger.error("Doh!", e);
    }
}

If does not fail when using Java w/ glibc. See attached log files with -Daws.crt.log.level=Debug enabled:

alpine-temurin21.log
ubuntu-temurin21.log

The used versions:

software.amazon.awssdk 2.24.10
software.amazon.awssdk.crt:aws-crt 0.29.13
Eclipse Temurin Container: eclipse-temurin:21-jre/eclipse-temurin:21-jre-alpine

Also, it does not fail if AwsCrtAsyncHttpClient is not used.

UditMishraMoodys · 2024-04-03T21:11:39Z

@debora-ito any update on this? we're facing the same issue with alpine and temurin 17 and it's blocking us

samleeflang · 2024-04-05T07:56:55Z

Got the same issue here as well, running on:

software.amazon.awssdk:s3 2.25.23
software.amazon.awssdk:aws-crt-client 2.25.23
runtime image eclipse-temurin:21-jre-alpine

Same issue running with the S3AsyncClient.builder().build() (no CRT) succeeds, running the CRT client it fails.
Giving exactly the same error messages as posted by agebhar1.

This reverts commit 26e1785. Hit an unexpected error that might be the same issue as this: aws/aws-sdk-java-v2#4935

KL0nLutiy · 2024-04-11T13:42:28Z

Same issue, using:

software.amazon.awssdk 2.25.18
software.amazon.awssdk.crt:aws-crt 0.29.13
runtime image amazoncorretto:11.0.20-alpine

vinujan59 · 2024-07-06T06:24:07Z

Changing the base image for the spring boot app from eclipse-temurin:17-jdk-alpine to eclipse-temurin:17-jdk-jammy works

software.amazon.awssdk 2.26.9
software.amazon.awssdk.crt:aws-crt 0.29.25

waahm7 · 2024-07-25T20:17:01Z

This should be fixed in https://github.com/awslabs/aws-crt-java/releases/tag/v0.30.3. Please try it out, and let me know if the issue persists.

ffried · 2024-07-26T06:16:52Z

@waahm7 I can confirm this to be fixed using the following versions:

software.amazon.aswssdk:bom -> 2.26.22
software.amazon.awssdk.crt:aws-crt -> 0.30.3

Thank you!

github-actions · 2024-07-31T17:45:17Z

This issue is now closed. Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.

ffried added bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels Feb 15, 2024

ffried mentioned this issue Feb 15, 2024

StackOverflowError on Alpine when using DefaultS3CrtAsyncClient awslabs/aws-crt-java#737

Closed

debora-ito added the p2 This is a standard priority issue label Feb 19, 2024

debora-ito self-assigned this Feb 19, 2024

debora-ito added investigating This issue is being investigated and/or work is in progress to resolve the issue. and removed needs-triage This issue or PR still needs to be triaged. labels Feb 19, 2024

debora-ito removed the investigating This issue is being investigated and/or work is in progress to resolve the issue. label Mar 29, 2024

rce added a commit to Opetushallitus/koodisto that referenced this issue Apr 9, 2024

Revert "OPHYK-2 Use S3CrtAsyncClient as recommended by AWS"

a13098b

This reverts commit 26e1785. Hit an unexpected error that might be the same issue as this: aws/aws-sdk-java-v2#4935

agebhar1 mentioned this issue May 24, 2024

Use official Apache Kafka image awslabs/eventbridge-kafka-connector#250

Merged

2 tasks

zoewangg mentioned this issue Jul 26, 2024

Bump CRT version to 0.30.3 #5433

Merged

zoewangg closed this as completed in #5433 Jul 31, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

StackOverflowError on Alpine when using DefaultS3CrtAsyncClient #4935

StackOverflowError on Alpine when using DefaultS3CrtAsyncClient #4935

ffried commented Feb 15, 2024

tedahn commented Feb 15, 2024 •

edited

Loading

debora-ito commented Feb 19, 2024 •

edited

Loading

JeHuiPark commented Mar 14, 2024

gweekleyprovidertrust commented Mar 14, 2024

agebhar1 commented Mar 22, 2024

UditMishraMoodys commented Apr 3, 2024 •

edited

Loading

samleeflang commented Apr 5, 2024 •

edited

Loading

KL0nLutiy commented Apr 11, 2024

vinujan59 commented Jul 6, 2024

waahm7 commented Jul 25, 2024

ffried commented Jul 26, 2024

github-actions bot commented Jul 31, 2024

StackOverflowError on Alpine when using DefaultS3CrtAsyncClient #4935

StackOverflowError on Alpine when using DefaultS3CrtAsyncClient #4935

Comments

ffried commented Feb 15, 2024

Describe the bug

Expected Behavior

Current Behavior

Reproduction Steps

Possible Solution

Additional Information/Context

AWS Java SDK version used

JDK version used

Operating System and version

tedahn commented Feb 15, 2024 • edited Loading

debora-ito commented Feb 19, 2024 • edited Loading

JeHuiPark commented Mar 14, 2024

gweekleyprovidertrust commented Mar 14, 2024

agebhar1 commented Mar 22, 2024

UditMishraMoodys commented Apr 3, 2024 • edited Loading

samleeflang commented Apr 5, 2024 • edited Loading

KL0nLutiy commented Apr 11, 2024

vinujan59 commented Jul 6, 2024

waahm7 commented Jul 25, 2024

ffried commented Jul 26, 2024

github-actions bot commented Jul 31, 2024

tedahn commented Feb 15, 2024 •

edited

Loading

debora-ito commented Feb 19, 2024 •

edited

Loading

UditMishraMoodys commented Apr 3, 2024 •

edited

Loading

samleeflang commented Apr 5, 2024 •

edited

Loading