StackOverflowError on Alpine when using DefaultS3CrtAsyncClient #737

ffried · 2023-12-20T17:27:59Z

Describe the bug

Using the DefaultS3CrtAsyncClient as produced by S3AsyncClient.crtBuilder().build() fails with a StackOverflowError on Alpine when running a ListObjectsRequest.

Using S3AsyncClient.builder().build() (no CRT) succeeds.

Expected Behavior

The CRT client works just as fine as the normal client (if not better 😉).

Current Behavior

A StackOverflowError is raised.

Reproduction Steps

Reproduction files:

Main.java

package de.sersoft;

import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.services.s3.S3AsyncClient;

public class Main {
    public static void main(String[] args) {
        var bucketName = args[0];
        try (var s3 = S3AsyncClient.crtBuilder().region(Region.US_EAST_1).build()) {
            System.out.println("Listing objects in bucket " + bucketName);
            var objects = s3.listObjects(builder -> builder.bucket(bucketName)).join().contents();
            System.out.println("Objects in bucket " + bucketName + ":");
            for (var myValue : objects) {
                System.out.println("-" + myValue.key());
            }
        }
    }
}

pom.xml

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>de.sersoft</groupId>
    <artifactId>AWSAlpine</artifactId>
    <version>1.0.0-SNAPSHOT</version>
    <packaging>jar</packaging>

    <properties>
        <maven.compiler.source>21</maven.compiler.source>
        <maven.compiler.target>21</maven.compiler.target>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    </properties>

    <dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>software.amazon.awssdk</groupId>
                <artifactId>bom</artifactId>
                <version>2.22.2</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
        </dependencies>
    </dependencyManagement>

    <dependencies>
        <dependency>
            <groupId>software.amazon.awssdk</groupId>
            <artifactId>s3</artifactId>
        </dependency>
        <dependency>
            <groupId>software.amazon.awssdk.crt</groupId>
            <artifactId>aws-crt</artifactId>
            <version>0.29.2</version>
        </dependency>
    </dependencies>

    <build>
        <finalName>${project.artifactId}</finalName>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-assembly-plugin</artifactId>
                <version>3.6.0</version>
                <executions>
                    <execution>
                        <phase>package</phase>
                        <goals><goal>single</goal></goals>
                        <configuration>
                            <archive>
                                <manifest><mainClass>de.sersoft.Main</mainClass></manifest>
                            </archive>
                            <descriptorRefs>
                                <descriptorRef>jar-with-dependencies</descriptorRef>
                            </descriptorRefs>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>
</project>

Dockerfile

FROM --platform=$BUILDPLATFORM maven:3-eclipse-temurin-21-alpine as buildnode

WORKDIR /aws-alpine

COPY . .

RUN --mount=type=cache,target=/root/.m2/repository mvn clean package -DskipTests


FROM eclipse-temurin:21-jdk-alpine
WORKDIR /aws-alpine

COPY --from=buildnode /aws-alpine/target/AWSAlpine-jar-with-dependencies.jar ./

ENTRYPOINT ["java", "-jar", "/aws-alpine/AWSAlpine-jar-with-dependencies.jar" ]

Then building (with buildx) and running the container, passing a bucket name as command will result in a StackOverflowError.

Possible Solution

No response

Additional Information/Context

Using a non-alpine image (removing the -alpine suffixes in the build layer and run layer base image) does not result in the same exception.

aws-crt-java version used

0.29.2

Java version used

21.0.1

Operating System and version

Alpine Linux v3.18

The text was updated successfully, but these errors were encountered:

TingDaoK · 2023-12-21T00:07:46Z

Can you provide more what the error is and CRT logs?

To enable logs from CRT, please refer to https://github.com/awslabs/aws-crt-java#system-properties

ffried · 2023-12-21T06:56:38Z

Ah, sorry, completely forgot to attach them yesterday.
Note that I've been using a bucket in eu-central-1. I've also replaced the bucket name with TEST-BUCKET as well as replaced the obvious credentials I've found in the logs with ****.

Stack Trace (from stdout)

2023-12-21T06:50:27.806548589Z Exception in thread "main" java.util.concurrent.CompletionException: software.amazon.awssdk.core.exception.SdkClientException: Unable to execute HTTP request: null
2023-12-21T06:50:27.806597422Z 	at software.amazon.awssdk.utils.CompletableFutureUtils.errorAsCompletionException(CompletableFutureUtils.java:65)
2023-12-21T06:50:27.806692589Z 	at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncExecutionFailureExceptionReportingStage.lambda$execute$0(AsyncExecutionFailureExceptionReportingStage.java:51)
2023-12-21T06:50:27.806713506Z 	at java.base/java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:934)
2023-12-21T06:50:27.806720256Z 	at java.base/java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:911)
2023-12-21T06:50:27.806724006Z 	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
2023-12-21T06:50:27.806728672Z 	at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2194)
2023-12-21T06:50:27.806730464Z 	at software.amazon.awssdk.utils.CompletableFutureUtils.lambda$forwardExceptionTo$0(CompletableFutureUtils.java:79)
2023-12-21T06:50:27.806732089Z 	at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
2023-12-21T06:50:27.806772214Z 	at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
2023-12-21T06:50:27.806776714Z 	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
2023-12-21T06:50:27.806778464Z 	at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2194)
2023-12-21T06:50:27.806780172Z 	at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.maybeAttemptExecute(AsyncRetryableStage.java:103)
2023-12-21T06:50:27.806781922Z 	at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.maybeRetryExecute(AsyncRetryableStage.java:184)
2023-12-21T06:50:27.806783672Z 	at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.lambda$attemptExecute$1(AsyncRetryableStage.java:159)
2023-12-21T06:50:27.806785381Z 	at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
2023-12-21T06:50:27.806787006Z 	at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
2023-12-21T06:50:27.806806672Z 	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
2023-12-21T06:50:27.806820339Z 	at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2194)
2023-12-21T06:50:27.806826006Z 	at software.amazon.awssdk.utils.CompletableFutureUtils.lambda$forwardExceptionTo$0(CompletableFutureUtils.java:79)
2023-12-21T06:50:27.806828006Z 	at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
2023-12-21T06:50:27.806829797Z 	at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
2023-12-21T06:50:27.806831547Z 	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
2023-12-21T06:50:27.806833172Z 	at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2194)
2023-12-21T06:50:27.806834797Z 	at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.lambda$execute$0(MakeAsyncHttpRequestStage.java:108)
2023-12-21T06:50:27.806836506Z 	at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
2023-12-21T06:50:27.806838131Z 	at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
2023-12-21T06:50:27.806840422Z 	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
2023-12-21T06:50:27.807046089Z 	at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2194)
2023-12-21T06:50:27.807053172Z 	at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.completeResponseFuture(MakeAsyncHttpRequestStage.java:255)
2023-12-21T06:50:27.807056589Z 	at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.lambda$executeHttpRequest$3(MakeAsyncHttpRequestStage.java:167)
2023-12-21T06:50:27.807058547Z 	at java.base/java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:934)
2023-12-21T06:50:27.807060256Z 	at java.base/java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:911)
2023-12-21T06:50:27.807061839Z 	at java.base/java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:482)
2023-12-21T06:50:27.807063422Z 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
2023-12-21T06:50:27.807072839Z 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
2023-12-21T06:50:27.807074464Z 	at java.base/java.lang.Thread.run(Thread.java:1583)
2023-12-21T06:50:27.807076047Z Caused by: software.amazon.awssdk.core.exception.SdkClientException: Unable to execute HTTP request: null
2023-12-21T06:50:27.807077714Z 	at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:111)
2023-12-21T06:50:27.807080131Z 	at software.amazon.awssdk.core.exception.SdkClientException.create(SdkClientException.java:47)
2023-12-21T06:50:27.807094964Z 	at software.amazon.awssdk.core.internal.http.pipeline.stages.utils.RetryableStageHelper.setLastException(RetryableStageHelper.java:223)
2023-12-21T06:50:27.807097506Z 	at software.amazon.awssdk.core.internal.http.pipeline.stages.utils.RetryableStageHelper.setLastException(RetryableStageHelper.java:218)
2023-12-21T06:50:27.807099172Z 	at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.maybeRetryExecute(AsyncRetryableStage.java:182)
2023-12-21T06:50:27.807100839Z 	... 23 more
2023-12-21T06:50:27.807627297Z Caused by: java.lang.StackOverflowError
2023-12-21T06:50:27.807646922Z 	at java.base/java.lang.ClassLoader.defineClass1(Native Method)
2023-12-21T06:50:27.807649256Z 	at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1027)
2023-12-21T06:50:27.807650922Z 	at java.base/java.security.SecureClassLoader.defineClass(SecureClassLoader.java:150)
2023-12-21T06:50:27.807652547Z 	at java.base/jdk.internal.loader.BuiltinClassLoader.defineClass(BuiltinClassLoader.java:862)
2023-12-21T06:50:27.807654131Z 	at java.base/jdk.internal.loader.BuiltinClassLoader.findClassOnClassPathOrNull(BuiltinClassLoader.java:760)
2023-12-21T06:50:27.807655756Z 	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(BuiltinClassLoader.java:681)
2023-12-21T06:50:27.807657297Z 	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:639)
2023-12-21T06:50:27.807658881Z 	at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
2023-12-21T06:50:27.807660464Z 	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:526)
2023-12-21T06:50:27.807662006Z 	at software.amazon.awssdk.core.internal.handler.BaseAsyncClientHandler.lambda$new$0(BaseAsyncClientHandler.java:66)
2023-12-21T06:50:27.807663631Z 	at software.amazon.awssdk.core.internal.http.async.AsyncResponseHandler.lambda$prepare$0(AsyncResponseHandler.java:92)
2023-12-21T06:50:27.807761047Z 	at java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1150)
2023-12-21T06:50:27.807773339Z 	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
2023-12-21T06:50:27.807775422Z 	at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2179)
2023-12-21T06:50:27.807777172Z 	at software.amazon.awssdk.core.internal.http.async.AsyncResponseHandler$BaosSubscriber.onComplete(AsyncResponseHandler.java:135)
2023-12-21T06:50:27.807778839Z 	at software.amazon.awssdk.core.internal.metrics.BytesReadTrackingPublisher$BytesReadTracker.onComplete(BytesReadTrackingPublisher.java:74)
2023-12-21T06:50:27.807780506Z 	at software.amazon.awssdk.utils.async.SimplePublisher.doProcessQueue(SimplePublisher.java:275)
2023-12-21T06:50:27.807782089Z 	at software.amazon.awssdk.utils.async.SimplePublisher.processEventQueue(SimplePublisher.java:224)
2023-12-21T06:50:27.807789631Z 	at software.amazon.awssdk.utils.async.SimplePublisher.complete(SimplePublisher.java:157)
2023-12-21T06:50:27.807791297Z 	at software.amazon.awssdk.services.s3.internal.crt.S3CrtResponseHandlerAdapter.onSuccessfulResponseComplete(S3CrtResponseHandlerAdapter.java:106)
2023-12-21T06:50:27.807792964Z 	at software.amazon.awssdk.services.s3.internal.crt.S3CrtResponseHandlerAdapter.onFinished(S3CrtResponseHandlerAdapter.java:101)
2023-12-21T06:50:27.807794589Z 	at software.amazon.awssdk.crt.s3.S3MetaRequestResponseHandlerNativeAdapter.onFinished(S3MetaRequestResponseHandlerNativeAdapter.java:25)

aws-crt.log

dzhus · 2024-01-11T20:42:34Z

2023-12-21T06:50:27.807627297Z Caused by: java.lang.StackOverflowError
2023-12-21T06:50:27.807646922Z at java.base/java.lang.ClassLoader.defineClass1(Native Method)
2023-12-21T06:50:27.807649256Z at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1027)
2023-12-21T06:50:27.807650922Z at java.base/java.security.SecureClassLoader.defineClass(SecureClassLoader.java:150)
2023-12-21T06:50:27.807652547Z at java.base/jdk.internal.loader.BuiltinClassLoader.defineClass(BuiltinClassLoader.java:862)
2023-12-21T06:50:27.807654131Z at java.base/jdk.internal.loader.BuiltinClassLoader.findClassOnClassPathOrNull(BuiltinClassLoader.java:760)
2023-12-21T06:50:27.807655756Z at java.base/jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(BuiltinClassLoader.java:681)
2023-12-21T06:50:27.807657297Z at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:639)
2023-12-21T06:50:27.807658881Z at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
2023-12-21T06:50:27.807660464Z at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:526)
2023-12-21T06:50:27.807662006Z at software.amazon.awssdk.core.internal.handler.BaseAsyncClientHandler.lambda$new$0(BaseAsyncClientHandler.java:66)

As a datapoint, I'm observing the same issue when using

FROM eclipse-temurin:11-jre-alpine@sha256:1cc1ad1955d30f45362175ae99414a2fc632383cefd2d626f823ec6d56523ec3

and it goes away when using eclipse-temurin:11-jre.
This is with software.amazon.awssdk.crt/aws-crt-0.29.6 and software.amazon.awssdk/s3-2.22.13.

tedahn · 2024-02-14T21:11:15Z

Hey, I'm also seeing the similar issues here. Just wanted to raise it as a +1

I'm using the S3TransferManager:

S3TransferManager.builder()
                .s3Client(S3AsyncClient.crtBuilder()
                        .httpConfiguration(config -> config.proxyConfiguration(s3CrtConfiguration))
                        .credentialsProvider(awsCredentialsProvider)
                        .region(Region.of(region))
                        .build())
                .build();

but we're seeing the error log

Caused by: java.lang.StackOverflowError: null

We are using it to execute a putObject but we're running into it before the connection happens.

And the image we are using is some modified version of: amazoncorretto:17 jdk-alpine 3.17

TingDaoK · 2024-02-14T23:21:36Z

Hi, I looked into the issue before, and I think it's probably an issue from aws-sdk-java-v2 instead. I did reach out to them, but I guess the conversation was lost. Can you create the issue at https://github.com/aws/aws-sdk-java-v2, so that it will be tracked and triaged better.

ffried · 2024-02-15T06:54:08Z

@TingDaoK Since no exception is thrown when using the default (non-crt) client, I'm not sure whether the issue is with the SDK. But if it helps to get more eyes on this to get to the bottom of it, copying this issue there is fine by me.
See aws/aws-sdk-java-v2#4935

TingDaoK · 2024-02-16T16:52:26Z

yeah, but from the stack trace we can see the exception comes from the sdk level, and also the crt log shows everything succeed. So, my current assumption is something went wrong about the integration between aws-sdk-java-v2 and aws-crt.

It only happens on alpine is also very interested and likely crt related. However, given the current logs and stack trace, I think we should start from the sdk side.

dzhus · 2024-02-27T10:44:23Z

The stack trace looks sliiightly different with

software.amazon.awssdk.crt/aws-crt-0.29.10
software.amazon.awssdk/s3-2.24.10,

now java.net.URLStreamHandler.setURL(Unknown Source) in the head.

Details

java.lang.StackOverflowError: null
	at java.net.URLStreamHandler.setURL(Unknown Source)
	at java.net.URLStreamHandler.setURL(Unknown Source)
	at sun.net.www.protocol.jar.Handler.parseURL(Unknown Source)
	at java.net.URL.<init>(Unknown Source)
	at java.net.URL.<init>(Unknown Source)
	at jdk.internal.loader.URLClassPath$JarLoader.checkResource(Unknown Source)
	at jdk.internal.loader.URLClassPath$JarLoader.getResource(Unknown Source)
	at jdk.internal.loader.URLClassPath.getResource(Unknown Source)
	at jdk.internal.loader.BuiltinClassLoader.findClassOnClassPathOrNull(Unknown Source)
	at jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(Unknown Source)
	at jdk.internal.loader.BuiltinClassLoader.loadClass(Unknown Source)
	at jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(Unknown Source)
	at java.lang.ClassLoader.loadClass(Unknown Source)
	at java.lang.ClassLoader.defineClass1(Unknown Source)
	at java.lang.ClassLoader.defineClass(Unknown Source)
	at java.security.SecureClassLoader.defineClass(Unknown Source)
	at jdk.internal.loader.BuiltinClassLoader.defineClass(Unknown Source)
	at jdk.internal.loader.BuiltinClassLoader.findClassOnClassPathOrNull(Unknown Source)
	at jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(Unknown Source)
	at jdk.internal.loader.BuiltinClassLoader.loadClass(Unknown Source)
	at jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(Unknown Source)
	at java.lang.ClassLoader.loadClass(Unknown Source)
	at java.lang.ClassLoader.defineClass1(Unknown Source)
	at java.lang.ClassLoader.defineClass(Unknown Source)
	at java.security.SecureClassLoader.defineClass(Unknown Source)
	at jdk.internal.loader.BuiltinClassLoader.defineClass(Unknown Source)
	at jdk.internal.loader.BuiltinClassLoader.findClassOnClassPathOrNull(Unknown Source)
	at jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(Unknown Source)
	at jdk.internal.loader.BuiltinClassLoader.loadClass(Unknown Source)
	at jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(Unknown Source)
	at java.lang.ClassLoader.loadClass(Unknown Source)
	at software.amazon.awssdk.core.internal.handler.BaseAsyncClientHandler.lambda$new$0(BaseAsyncClientHandler.java:66)
	at software.amazon.awssdk.core.internal.http.async.AsyncResponseHandler.lambda$prepare$0(AsyncResponseHandler.java:92)
	at java.util.concurrent.CompletableFuture$UniCompose.tryFire(Unknown Source)
	at java.util.concurrent.CompletableFuture.postComplete(Unknown Source)
	at java.util.concurrent.CompletableFuture.complete(Unknown Source)
	at software.amazon.awssdk.core.internal.http.async.AsyncResponseHandler$BaosSubscriber.onComplete(AsyncResponseHandler.java:135)
	at software.amazon.awssdk.core.internal.metrics.BytesReadTrackingPublisher$BytesReadTracker.onComplete(BytesReadTrackingPublisher.java:74)
	at software.amazon.awssdk.utils.async.SimplePublisher.doProcessQueue(SimplePublisher.java:275)
	at software.amazon.awssdk.utils.async.SimplePublisher.processEventQueue(SimplePublisher.java:224)
	at software.amazon.awssdk.utils.async.SimplePublisher.complete(SimplePublisher.java:157)
	at software.amazon.awssdk.services.s3.internal.crt.S3CrtResponseHandlerAdapter.onSuccessfulResponseComplete(S3CrtResponseHandlerAdapter.java:140)
	at software.amazon.awssdk.services.s3.internal.crt.S3CrtResponseHandlerAdapter.onFinished(S3CrtResponseHandlerAdapter.java:135)
	at software.amazon.awssdk.crt.s3.S3MetaRequestResponseHandlerNativeAdapter.onFinished(S3MetaRequestResponseHandlerNativeAdapter.java:25)
Wrapped by: software.amazon.awssdk.core.exception.SdkClientException: Unable to execute HTTP request: null
	at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:111)
	at software.amazon.awssdk.core.exception.SdkClientException.create(SdkClientException.java:47)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.utils.RetryableStageHelper.setLastException(RetryableStageHelper.java:223)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.utils.RetryableStageHelper.setLastException(RetryableStageHelper.java:218)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.maybeRetryExecute(AsyncRetryableStage.java:182)
	... 23 common frames omitted
Wrapped by: java.util.concurrent.CompletionException: software.amazon.awssdk.core.exception.SdkClientException: Unable to execute HTTP request: null
	at software.amazon.awssdk.utils.CompletableFutureUtils.errorAsCompletionException(CompletableFutureUtils.java:65)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncExecutionFailureExceptionReportingStage.lambda$execute$0(AsyncExecutionFailureExceptionReportingStage.java:51)
	at java.util.concurrent.CompletableFuture.uniHandle(Unknown Source)
	at java.util.concurrent.CompletableFuture$UniHandle.tryFire(Unknown Source)
	at java.util.concurrent.CompletableFuture.postComplete(Unknown Source)
	at java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source)
	at software.amazon.awssdk.utils.CompletableFutureUtils.lambda$forwardExceptionTo$0(CompletableFutureUtils.java:79)
	at java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown Source)
	at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(Unknown Source)
	at java.util.concurrent.CompletableFuture.postComplete(Unknown Source)
	at java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.maybeAttemptExecute(AsyncRetryableStage.java:103)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.maybeRetryExecute(AsyncRetryableStage.java:184)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.lambda$attemptExecute$1(AsyncRetryableStage.java:159)
	at java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown Source)
	at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(Unknown Source)
	at java.util.concurrent.CompletableFuture.postComplete(Unknown Source)
	at java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source)
	at software.amazon.awssdk.utils.CompletableFutureUtils.lambda$forwardExceptionTo$0(CompletableFutureUtils.java:79)
	at java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown Source)
	at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(Unknown Source)
	at java.util.concurrent.CompletableFuture.postComplete(Unknown Source)
	at java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.lambda$execute$0(MakeAsyncHttpRequestStage.java:108)
	at java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown Source)
	at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(Unknown Source)
	at java.util.concurrent.CompletableFuture.postComplete(Unknown Source)
	at java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.completeResponseFuture(MakeAsyncHttpRequestStage.java:255)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.lambda$executeHttpRequest$3(MakeAsyncHttpRequestStage.java:167)
	at java.util.concurrent.CompletableFuture.uniHandle(Unknown Source)
	at java.util.concurrent.CompletableFuture$UniHandle.tryFire(Unknown Source)
	at java.util.concurrent.CompletableFuture$Completion.run(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.lang.Thread.run(Unknown Source)

agebhar1 · 2024-03-22T17:30:01Z

See also aws/aws-sdk-java-v2#4935 (comment) with attached log files with -Daws.crt.log.level=Debug enabled.

The used versions:

software.amazon.awssdk 2.24.10
software.amazon.awssdk.crt:aws-crt 0.29.13
Eclipse Temurin Container: eclipse-temurin:21-jre/eclipse-temurin:21-jre-alpine

UditMishraMoodys · 2024-04-03T21:11:20Z

Any update on this?

renegrob · 2024-06-18T06:14:00Z

I encountered the same issue with:

software.amazon.awssdk:2.26.3
software.amazon.awssdk.crt:aws-crt:0.29.23

waahm7 · 2024-06-21T21:58:05Z

Sorry for the late response. Can someone please share your current thread stack size and try increasing it? From the logs, we don’t see any errors, so I suspect it might be a smaller stack size on Alpine.

agebhar1 · 2024-07-21T07:11:54Z

@waahm7 see trace (-Daws.crt.memory.tracing=2 -Daws.crt.log.level=Trace -Daws.crt.log.destination=Stderr) for Alpine error here alpine.log. It's created with this minimum project setup aws-crt-java-gh737.

waahm7 · 2024-07-21T15:29:30Z

@agebhar1 Can you please try increasing the thread stack size by setting the PTHREAD_STACK_MIN environment variable to 8388608 (8 MB)?

waahm7 · 2024-07-21T15:49:53Z

@agebhar1 Thanks for the project. I have tried running it and getting

docker.errors.DockerException: Error while fetching server API version: HTTPConnection.request() got an unexpected keyword argument 'chunked'

Is there anything else I am supposed to setup?

agebhar1 · 2024-07-21T16:46:01Z

I did @waahm7 inside the Alpine container via:

$ PTHREAD_STACK_MIN=8388608 java -jar -Daws.crt.memory.tracing=2 -Daws.crt.log.level=Trace -Daws.crt.log.destination=File -Daws.crt.log.filename=xyz /mnt/aws-crt-java-with-dependencies.jar

but still the java.lang.StackOverflowError.

agebhar1 · 2024-07-21T16:46:44Z

@agebhar1 Thanks for the project. I have tried running it and getting
docker.errors.DockerException: Error while fetching server API version: HTTPConnection.request() got an unexpected keyword argument 'chunked'
Is there anything else I am supposed to setup?

Which version of Docker and Docker Compose and OS do you use?

agebhar1 · 2024-07-21T17:06:07Z

The definition of PTHREAD_STACK_MIN from http://git.musl-libc.org/cgit/musl/tree/include/limits.h#n67 is 2kb (2048) and seems not to be configurable. I did a grep on MUSL 1.2.5:

musl-1.2.5 $ rg PTHREAD_STACK_MIN
include/limits.h
67:#define PTHREAD_STACK_MIN 2048

src/conf/sysconf.c
91:             [_SC_THREAD_STACK_MIN] = PTHREAD_STACK_MIN,

src/thread/pthread_attr_setstacksize.c
5:      if (size-PTHREAD_STACK_MIN > SIZE_MAX/4) return EINVAL;

src/thread/pthread_attr_setstack.c
5:      if (size-PTHREAD_STACK_MIN > SIZE_MAX/4) return EINVAL;

waahm7 · 2024-07-23T01:51:17Z

Thanks, I have verified that the issue is the low thread stack size on Musl. I was able to reproduce the issue and was able to fix it by bumping the thread stack size to 2MiB. I am looking into a potential fix for this issue.

agebhar1 · 2024-07-23T17:16:30Z

Great news @waahm7 👍 How did you increased the thread stack size? I tried with the environment variable PTHREAD_STACK_MIN without success.

waahm7 · 2024-07-23T21:31:21Z

@agebhar1, Thank you for your help and the sample. I hacked our low-level code to increase the stack size using pthread_attr_setstacksize. I am working on creating a pull request to bump the stack size to at least 1MiB if it's less than that.

waahm7 · 2024-07-25T20:02:45Z

This should be fixed in https://github.com/awslabs/aws-crt-java/releases/tag/v0.30.3. Please try it out, and let me know if the issue persists.

ffried · 2024-07-26T06:17:15Z

@waahm7 I can confirm this to be fixed using the following versions:

software.amazon.aswssdk:bom -> 2.26.22
software.amazon.awssdk.crt:aws-crt -> 0.30.3

Thank you!

agebhar1 · 2024-07-26T15:51:27Z

I can confirm it too @waahm7 for the sample 👍

ffried added bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels Dec 20, 2023

jmklix added investigating This issue is being investigated and/or work is in progress to resolve the issue. p2 This is a standard priority issue and removed needs-triage This issue or PR still needs to be triaged. labels Dec 27, 2023

ffried mentioned this issue Feb 15, 2024

StackOverflowError on Alpine when using DefaultS3CrtAsyncClient aws/aws-sdk-java-v2#4935

Closed

This was referenced Jul 23, 2024

Bump the minimum stack size to at least 1MB awslabs/aws-c-common#1139

Merged

Bump the minimum stack size to at least 1MB #809

Merged

waahm7 added response-requested Waiting on additional info and feedback. Will move to 'closing-soon' in 7 days. and removed investigating This issue is being investigated and/or work is in progress to resolve the issue. labels Jul 25, 2024

github-actions bot removed the response-requested Waiting on additional info and feedback. Will move to 'closing-soon' in 7 days. label Jul 26, 2024

waahm7 closed this as completed Jul 26, 2024

agebhar1 mentioned this issue Jul 27, 2024

Use official Apache Kafka image awslabs/eventbridge-kafka-connector#250

Merged

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

StackOverflowError on Alpine when using DefaultS3CrtAsyncClient #737

StackOverflowError on Alpine when using DefaultS3CrtAsyncClient #737

ffried commented Dec 20, 2023 •

edited

Loading

TingDaoK commented Dec 21, 2023

ffried commented Dec 21, 2023

dzhus commented Jan 11, 2024

tedahn commented Feb 14, 2024

TingDaoK commented Feb 14, 2024

ffried commented Feb 15, 2024 •

edited

Loading

TingDaoK commented Feb 16, 2024 •

edited

Loading

dzhus commented Feb 27, 2024

agebhar1 commented Mar 22, 2024

UditMishraMoodys commented Apr 3, 2024

renegrob commented Jun 18, 2024

waahm7 commented Jun 21, 2024 •

edited

Loading

agebhar1 commented Jul 21, 2024

waahm7 commented Jul 21, 2024 •

edited

Loading

waahm7 commented Jul 21, 2024

agebhar1 commented Jul 21, 2024 •

edited

Loading

agebhar1 commented Jul 21, 2024

agebhar1 commented Jul 21, 2024

waahm7 commented Jul 23, 2024

agebhar1 commented Jul 23, 2024

waahm7 commented Jul 23, 2024 •

edited

Loading

waahm7 commented Jul 25, 2024

ffried commented Jul 26, 2024

agebhar1 commented Jul 26, 2024

StackOverflowError on Alpine when using DefaultS3CrtAsyncClient #737

StackOverflowError on Alpine when using DefaultS3CrtAsyncClient #737

Comments

ffried commented Dec 20, 2023 • edited Loading

Describe the bug

Expected Behavior

Current Behavior

Reproduction Steps

Possible Solution

Additional Information/Context

aws-crt-java version used

Java version used

Operating System and version

TingDaoK commented Dec 21, 2023

ffried commented Dec 21, 2023

dzhus commented Jan 11, 2024

tedahn commented Feb 14, 2024

TingDaoK commented Feb 14, 2024

ffried commented Feb 15, 2024 • edited Loading

TingDaoK commented Feb 16, 2024 • edited Loading

dzhus commented Feb 27, 2024

agebhar1 commented Mar 22, 2024

UditMishraMoodys commented Apr 3, 2024

renegrob commented Jun 18, 2024

waahm7 commented Jun 21, 2024 • edited Loading

agebhar1 commented Jul 21, 2024

waahm7 commented Jul 21, 2024 • edited Loading

waahm7 commented Jul 21, 2024

agebhar1 commented Jul 21, 2024 • edited Loading

agebhar1 commented Jul 21, 2024

agebhar1 commented Jul 21, 2024

waahm7 commented Jul 23, 2024

agebhar1 commented Jul 23, 2024

waahm7 commented Jul 23, 2024 • edited Loading

waahm7 commented Jul 25, 2024

ffried commented Jul 26, 2024

agebhar1 commented Jul 26, 2024

ffried commented Dec 20, 2023 •

edited

Loading

ffried commented Feb 15, 2024 •

edited

Loading

TingDaoK commented Feb 16, 2024 •

edited

Loading

waahm7 commented Jun 21, 2024 •

edited

Loading

waahm7 commented Jul 21, 2024 •

edited

Loading

agebhar1 commented Jul 21, 2024 •

edited

Loading

waahm7 commented Jul 23, 2024 •

edited

Loading