Parallel split for multipart GetObject File Download #6425

L-Applin · 2025-09-16T15:57:22Z

Implement parallel download for multipart GetObject in s3 Async Client and Transfer Manager.

Modifications

Add two new classes (Publisher/Subscriber) to orchestrate the non-linear multipart download: NonLinearMultipartDownloaderSubscriber and FileAsyncResponseTransformerPublisher. Note for reviewer: This is the core of the PR new functionality and review should probably start with those two classes.
Add support in Transfer-Manager module for Transfer Progress Updater.
- Note for reviewer: The AsyncResponseTransformer published by FileAsyncResponseTransformerPublisher needs to wrapped to publish progress to the progress updater. This is done in GenericS3TransferManager and TransferProgressUpdater
New public API, as discussed during design review
- supportNonSerial on SplitResult
- ParallelConfiguration new config class in MultipartConfiguration for the maxInFlightParts config
New internal API
- FileAsyncTransformer exposes getters for position, path and FileTransformerConfiguration

Testing

Added unit test
Added integration test
Manual tests using large objects

…in the onResponse callback. Keep track of all inflight requests.

…riber

…id-multi

...src/main/java/software/amazon/awssdk/core/internal/async/ThreadSafeEmittingSubscription.java

utils/src/main/java/software/amazon/awssdk/utils/ContentRangeParser.java

zoewangg

Still going through the PR.

core/sdk-core/src/main/java/software/amazon/awssdk/core/async/AsyncResponseTransformer.java

...e/src/main/java/software/amazon/awssdk/core/internal/async/FileAsyncResponseTransformer.java

...src/main/java/software/amazon/awssdk/core/internal/async/ThreadSafeEmittingSubscription.java

...e/src/main/java/software/amazon/awssdk/core/internal/async/FileAsyncResponseTransformer.java

...are/amazon/awssdk/services/s3/internal/multipart/NonLinearMultipartDownloaderSubscriber.java

...n/java/software/amazon/awssdk/core/internal/async/FileAsyncResponseTransformerPublisher.java

core/sdk-core/src/main/java/software/amazon/awssdk/core/async/AsyncResponseTransformer.java

...n/java/software/amazon/awssdk/core/internal/async/FileAsyncResponseTransformerPublisher.java

...are/amazon/awssdk/services/s3/internal/multipart/NonLinearMultipartDownloaderSubscriber.java

…blisher

…ents - try/catch in ThreadSafeEmittingSubscription - negative totalParts check - NonLinearMultipartDownloaderSubscriberTckTest tck test

core/sdk-core/src/main/java/software/amazon/awssdk/core/async/AsyncResponseTransformer.java

...a/software/amazon/awssdk/core/internal/async/DefaultAsyncResponseTransformerSplitResult.java

...n/java/software/amazon/awssdk/core/internal/async/FileAsyncResponseTransformerPublisher.java

...src/main/java/software/amazon/awssdk/core/internal/async/ThreadSafeEmittingSubscription.java

...are/amazon/awssdk/services/s3/internal/multipart/NonLinearMultipartDownloaderSubscriber.java

joviegas · 2025-09-18T20:07:48Z

...n/java/software/amazon/awssdk/core/internal/async/FileAsyncResponseTransformerPublisher.java

+
+        @Override
+        public void onResponse(T response) {
+            Optional<String> contentRangeList = response.sdkHttpResponse().firstMatchingHeader("x-amz-content-range");


Since we are depending on header sent by service will this cause issue for Third party tools like minio or gcp ? Since we are erring out if the header is not present it would be good to know its impact when used with Third party s3 like minio or gcp

...va/software/amazon/awssdk/services/s3/internal/multipart/MultipartConfigurationResolver.java

...are/amazon/awssdk/services/s3/internal/multipart/NonLinearMultipartDownloaderSubscriber.java

- renamed EmittingSubscription, mark it ThreadSafe - Added comments - some other renaming

...are/amazon/awssdk/services/s3/internal/multipart/NonLinearMultipartDownloaderSubscriber.java

...ware/amazon/awssdk/services/s3/internal/multipart/ParallelMultipartDownloaderSubscriber.java

...are/amazon/awssdk/services/s3/internal/multipart/NonLinearMultipartDownloaderSubscriber.java

core/sdk-core/src/main/java/software/amazon/awssdk/core/async/AsyncResponseTransformer.java

.../sdk-core/src/main/java/software/amazon/awssdk/core/internal/async/EmittingSubscription.java

alextwoods · 2025-09-23T16:04:08Z

...n/java/software/amazon/awssdk/core/internal/async/FileAsyncResponseTransformerPublisher.java

+
+        @Override
+        public void onResponse(T response) {
+            Optional<String> contentRangeList = response.sdkHttpResponse().firstMatchingHeader("x-amz-content-range");


Is this logic specific to S3? Could we possibly apply it to a generic streaming service? In general my guess is no, because other streaming APIs don't necessarily support content-range (at least my cursory inspection most I looked at do not support requests or responses with content-range).

Given that - should we keep this class in S3 instead of core?

...ware/amazon/awssdk/services/s3/internal/multipart/ParallelMultipartDownloaderSubscriber.java

...n/java/software/amazon/awssdk/core/internal/async/FileAsyncResponseTransformerPublisher.java

dagnir · 2025-09-23T18:46:55Z

...n/java/software/amazon/awssdk/core/internal/async/FileAsyncResponseTransformerPublisher.java

+            CompletableFuture<T> delegateFuture = delegate.prepare();
+            CompletableFutureUtils.forwardResultTo(delegateFuture, future);
+            CompletableFutureUtils.forwardExceptionTo(future, delegateFuture);


Seems like this should be inverted so that IndividualTransformer does the exception/result forwarding within prepare. That would clean things up a little

We can't call delegate.prepare() in the actual prepare() callback of this IndividualFileTransformer because we need to have received the SdkResponse to create the delegate, to know at which offset to write to (after reading the content range header). This is why delegate.prepare() is called here in onResponse

...n/java/software/amazon/awssdk/core/internal/async/FileAsyncResponseTransformerPublisher.java

.../sdk-core/src/main/java/software/amazon/awssdk/core/internal/async/EmittingSubscription.java

...e/src/main/java/software/amazon/awssdk/core/internal/async/FileAsyncResponseTransformer.java

...are/amazon/awssdk/services/s3/internal/multipart/NonLinearMultipartDownloaderSubscriber.java

...ware/amazon/awssdk/services/s3/internal/multipart/ParallelMultipartDownloaderSubscriber.java

...are/amazon/awssdk/services/s3/internal/multipart/NonLinearMultipartDownloaderSubscriber.java

…oposal. Renamed to ParallelMultipartDownloaderSubscriber as per PR comment - Other PR comment: Removed unused builder parameter for EmittingSubscription

…n/large-object-merge # Conflicts: # services/s3/src/test/java/software/amazon/awssdk/services/s3/internal/multipart/S3MultipartFileDownloadWiremockTest.java

...ware/amazon/awssdk/services/s3/internal/multipart/ParallelMultipartDownloaderSubscriber.java

sonarqubecloud · 2025-09-25T19:17:36Z

Quality Gate failed

Failed conditions
66.2% Coverage on New Code (required ≥ 80%)

See analysis details on SonarQube Cloud

L-Applin added 28 commits July 22, 2025 18:15

exploration

c5ed15e

hide cloudwatch logs for s3 regression tests

067db30

Merge remote-tracking branch 'origin/master'

41be0a6

Multipart dowload implementation

51357ee

Multipart dowload implementation

a9d25ec

Merge remote-tracking branch 'origin/master'

b70ab8b

Merge remote-tracking branch 'origin/master'

0a8f6c5

Update non-linear download logic

7c82e2b

request max-in-flight at first request

978f553

remove stateful calls to FileAsyncResponseTransformer by creating it …

3e29b8c

…in the onResponse callback. Keep track of all inflight requests.

Wiremock test for s3AsyncClient with AsyncResponseTransformer.toFile

c44a520

added 10_000 parts tests

418dcbd

add missing assertions to errorOnMiddlePart_retryable

db7d301

handle fileWriteOption

2f65a8c

prevent over-requesting and fix potential race condition in FileSubsc…

0bfbdb2

…riber

TM integration

0a20a9c

fix tests

e1aebb5

debugging for long stop before completing future

3a0a0b1

added outstanding demand tracking, assert with checksum in integ test

2f3a14e

Merge remote-tracking branch 'origin/master'

e49aadc

Fix pendingTransformer polling

aa0caf1

Make download method of TM work properly with progressUpdater

29f899d

remove logs

0aa1529

add ParallelConfiguration for maxInFlightParts

426d82d

javadoc and logs

5bdc142

Merge remote-tracking branch 'origin/master'

9551671

Merge remote-tracking branch 'origin/master' into feature/master/hagr…

7e19d14

…id-multi

Merge branch 'master' into feature/master/hagrid-multi

d37056d

L-Applin commented Sep 16, 2025

View reviewed changes

...src/main/java/software/amazon/awssdk/core/internal/async/ThreadSafeEmittingSubscription.java Show resolved Hide resolved

L-Applin commented Sep 16, 2025

View reviewed changes

utils/src/main/java/software/amazon/awssdk/utils/ContentRangeParser.java Show resolved Hide resolved

zoewangg reviewed Sep 17, 2025

View reviewed changes

S-Saranya1 reviewed Sep 17, 2025

View reviewed changes

...are/amazon/awssdk/services/s3/internal/multipart/NonLinearMultipartDownloaderSubscriber.java Show resolved Hide resolved

L-Applin added 5 commits September 17, 2025 23:32

PR comments - first pass

1716e46

PR comments - keep transformerCount in FileAsyncResponseTransformerPu…

289554e

…blisher

PR comments - keep transformerCount in FileAsyncResponseTransformerPu…

e025fb3

…blisher

rename supportsNonSerial to parallelSplitSupported, and other PR comm…

fe7985f

…ents - try/catch in ThreadSafeEmittingSubscription - negative totalParts check - NonLinearMultipartDownloaderSubscriberTckTest tck test

Added file config options to FileAsyncResponseTransformerPublisherTest

af94ef1

joviegas reviewed Sep 18, 2025

View reviewed changes

L-Applin added 3 commits September 18, 2025 17:56

PR comments

9902ec8

- renamed EmittingSubscription, mark it ThreadSafe - Added comments - some other renaming

checkstyle

e6c824c

checkstyle

13098d3

zoewangg reviewed Sep 23, 2025

View reviewed changes

L-Applin added 2 commits September 23, 2025 10:52

fix file issues in NonLinearMultipartDownloaderSubscriberWiremockTest.

cfb68c0

try to fix Jimfs tests again

265c50c

alextwoods reviewed Sep 23, 2025

View reviewed changes

dagnir self-requested a review September 23, 2025 17:55

dagnir requested changes Sep 23, 2025

View reviewed changes

dagnir reviewed Sep 23, 2025

View reviewed changes

L-Applin added 6 commits September 24, 2025 11:22

- NonLinearMultipartDownloaderSubscriber refactor based on zoewang pr…

d9588d8

…oposal. Renamed to ParallelMultipartDownloaderSubscriber as per PR comment - Other PR comment: Removed unused builder parameter for EmittingSubscription

Merge branch 'olapplin/changes-to-non-linear-subscriber' into olappli…

cab2c1f

…n/large-object-merge # Conflicts: # services/s3/src/test/java/software/amazon/awssdk/services/s3/internal/multipart/S3MultipartFileDownloadWiremockTest.java

final ParallelMultipartDownloaderSubscriber fields

039c4ce

mark ContentRangeParser as protected API

d3054bd

checkstyle

732e337

use debug logs

1466a6e

dagnir reviewed Sep 24, 2025

View reviewed changes

...ware/amazon/awssdk/services/s3/internal/multipart/ParallelMultipartDownloaderSubscriber.java Outdated Show resolved Hide resolved

...ware/amazon/awssdk/services/s3/internal/multipart/ParallelMultipartDownloaderSubscriber.java Outdated Show resolved Hide resolved

use debug logs

ea2f6aa

L-Applin changed the title ~~Olapplin/large object merge~~ Parallel split for multipart GetObject File Download Sep 25, 2025

fix S3MultipartClientGetObjectWiremockTest

f2eae2e

Parallel split for multipart GetObject File Download #6425

Are you sure you want to change the base?

Parallel split for multipart GetObject File Download #6425

Uh oh!

Conversation

L-Applin commented Sep 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Modifications

Testing

Uh oh!

Uh oh!

Uh oh!

zoewangg left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

joviegas Sep 18, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

alextwoods Sep 23, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

dagnir Sep 23, 2025

Choose a reason for hiding this comment

Uh oh!

L-Applin Sep 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

L-Applin commented Sep 16, 2025 •

edited

Loading

L-Applin Sep 24, 2025 •

edited

Loading