AWS CRT-based S3 client 400 bad request during upload files to ceph.

[IgorekSergeevich]

Hello everyone, I encountered a problem when upload files to ceph, I use aws-crt async client. For some reason I get 400 bad request during upload (it's doesn't matter multipart upload or just upload). Upload to aws s3 and minio works fine. Maybe you have any ideas what it could be?

Dependencies which i use:

implementation "software.amazon.awssdk:s3:2.28.6"
implementation "software.amazon.awssdk.crt:aws-crt:0.33.9"

What i see in console:

software.amazon.awssdk.services.s3.model.S3Exception: (Service: S3, Status Code: 400, Request ID: tx00000e6ef5fccbf3b15a0-0067921b65-513bf62-ceph-objectstore)
	at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handleErrorResponse(AwsXmlPredicatedResponseHandler.java:156)
	Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException: 

Original Stack Trace:
		at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handleErrorResponse(AwsXmlPredicatedResponseHandler.java:156)
		at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handleResponse(AwsXmlPredicatedResponseHandler.java:108)
		at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handle(AwsXmlPredicatedResponseHandler.java:85)
		at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handle(AwsXmlPredicatedResponseHandler.java:43)
		at software.amazon.awssdk.core.internal.handler.BaseClientHandler.lambda$successTransformationResponseHandler$7(BaseClientHandler.java:279)
		at software.amazon.awssdk.core.internal.http.async.AsyncResponseHandler.lambda$prepare$0(AsyncResponseHandler.java:92)
		at java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1150)
		at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
		at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
		at software.amazon.awssdk.core.internal.http.async.AsyncResponseHandler$BaosSubscriber.onComplete(AsyncResponseHandler.java:135)
		at software.amazon.awssdk.core.internal.metrics.BytesReadTrackingPublisher$BytesReadTracker.onComplete(BytesReadTrackingPublisher.java:74)
		at software.amazon.awssdk.utils.async.SimplePublisher.doProcessQueue(SimplePublisher.java:275)
		at software.amazon.awssdk.utils.async.SimplePublisher.processEventQueue(SimplePublisher.java:224)
		at software.amazon.awssdk.utils.async.SimplePublisher.complete(SimplePublisher.java:157)
		at java.base/java.util.concurrent.CompletableFuture.uniRunNow(CompletableFuture.java:819)
		at java.base/java.util.concurrent.CompletableFuture.uniRunStage(CompletableFuture.java:803)
		at java.base/java.util.concurrent.CompletableFuture.thenRun(CompletableFuture.java:2195)
		at software.amazon.awssdk.services.s3.internal.crt.S3CrtResponseHandlerAdapter.onErrorResponseComplete(S3CrtResponseHandlerAdapter.java:200)
		at software.amazon.awssdk.services.s3.internal.crt.S3CrtResponseHandlerAdapter.handleError(S3CrtResponseHandlerAdapter.java:179)
		at software.amazon.awssdk.services.s3.internal.crt.S3CrtResponseHandlerAdapter.onFinished(S3CrtResponseHandlerAdapter.java:148)
		at software.amazon.awssdk.crt.s3.S3MetaRequestResponseHandlerNativeAdapter.onFinished(S3MetaRequestResponseHandlerNativeAdapter.java:25)

What i see in ceph log file:

"POST /customers-bucket/76/94/559476/bosch-annual-report-2023%20%281%29.pdf?uploads HTTP/1.1" 200 307 - "aws-sdk-java/2.28.6 md/io#async md/http#s3crt ua/2.0 os/Linux#5.15.0-60-generic lang/java#17.0.13 md/OpenJDK_64-Bit_Server_VM#17.0.13+12-LTS md/vendor#BellSoft md/en_US md/kotlin/1.9.21-release-633 cfg/retry-mode#standard cfg/auth-source#stat CRTS3NativeClient/0.1.x platform/unknown" - latency=0.010000149s
"PUT /customers-bucket/76/94/559476/bosch-annual-report-2023%20%281%29.pdf?partNumber=1&uploadId=2~WEWFO-TZ_90u9DJz8y7hXibTWR5n79o HTTP/1.1" 400 8388941 - "aws-sdk-java/2.28.6 md/io#async md/http#s3crt ua/2.0 os/Linux#5.15.0-60-generic lang/java#17.0.13 md/OpenJDK_64-Bit_Server_VM#17.0.13+12-LTS md/vendor#BellSoft md/en_US md/kotlin/1.9.21-release-633 cfg/retry-mode#standard cfg/auth-source#stat CRTS3NativeClient/0.1.x platform/unknown" - latency=0.134002000s
"PUT /customers-bucket/76/94/559476/bosch-annual-report-2023%20%281%29.pdf?partNumber=2&uploadId=2~WEWFO-TZ_90u9DJz8y7hXibTWR5n79o HTTP/1.1" 400 1700859 - "aws-sdk-java/2.28.6 md/io#async md/http#s3crt ua/2.0 os/Linux#5.15.0-60-generic lang/java#17.0.13 md/OpenJDK_64-Bit_Server_VM#17.0.13+12-LTS md/vendor#BellSoft md/en_US md/kotlin/1.9.21-release-633 cfg/retry-mode#standard cfg/auth-source#stat CRTS3NativeClient/0.1.x platform/unknown" - latency=0.018000269s

My kotlin spring config for client:

import java.net.URI
import org.springframework.context.annotation.Bean
import org.springframework.context.annotation.Configuration
import software.amazon.awssdk.auth.credentials.AwsBasicCredentials
import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider
import software.amazon.awssdk.awscore.retry.AwsRetryStrategy
import software.amazon.awssdk.http.crt.AwsCrtAsyncHttpClient
import software.amazon.awssdk.services.s3.S3AsyncClient

@Bean
fun s3Client(properties: S3Properties): S3AsyncClient {
    val credentials = AwsBasicCredentials.create(properties.accessKey, properties.secretKey)
    
    val httpClient = AwsCrtAsyncHttpClient.builder()
        .proxyConfiguration { proxyConfig ->
            proxyConfig
                .useEnvironmentVariableValues(false)
                .useSystemPropertyValues(false)
        }
        .build()
    
    val retryStrategy = AwsRetryStrategy.standardRetryStrategy()
        .toBuilder()
        .maxAttempts(properties.retryMaxAttempts)
        .build()

    return S3AsyncClient.builder()
        .endpointOverride(URI(properties.endpoint))
        .credentialsProvider(StaticCredentialsProvider.create(credentials))
        .forcePathStyle(properties.forcePathStyle)
        .httpClient(httpClient)
        .region(properties.region)
        .overrideConfiguration { configuration ->
            configuration.retryStrategy(retryStrategy)
        }
        .build()
}

[debora-ito]

Hi @IgorekSergeevich you need to reach out to the Ceph support team for help with troubleshooting. We don't guarantee that the AWS SDKs will work with third-party s3-compatible solutions.

[steveloughran]

Looks like whatever got sent back was rejected.

1. Cut out the CRT & see if it is happy
2. Download hadoop 3.4.1 binaries and try to upload from there -small file then something > 100 MB
3. If that doesn't work, file a JIRA on the apache JIRA server. It's not something we've tested ourselves -but third party store support does matter to us.

Out the box, the SDK is not that good at recovering from failures when uploading a part of a multi part upload, and a 400 error was where it surfaced -though that was not the root cause -which for us was a rare transient failure in AWS S3 itself.

Given you are seeing this on every upload attempt it is probably something different – but would be interesting to see what