Refactor S3 Transfer Manager uploadDirectory to limit the number of c…

[zoewangg]

…oncurrent upload file requests.

Motivation and Context

Currently, there's no limit on the number of uploadFileRequets in parallel; it depends on how fast it traverses the file and all futures are kept in memory. This could cause memory issues for users who upload a directory that has millions of files because millions of futures will be in memory until upload directory finishes.
See #5023

Modifications

• Set a limit on the number of concurrent upload file requests for upload directory by using AsyncBufferingSubscriber that controls the request concurrency (it's used in downloadDirectory as well).

• Only keep the request futures that are in flight by not storing all futures in a list and not keeping reference through CompletableFutureUtils.forwardExceptionTo

Testing

Existing tests passed.
Tested on EC2 instance to upload a large directory and verified memory usage.

Screenshots (if appropriate)

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)

Checklist

• I have read the CONTRIBUTING document
• Local run of mvn install succeeds
• My code follows the code style of this project
• My change requires a change to the Javadoc documentation
• I have updated the Javadoc documentation accordingly
• I have added tests to cover my changes
• All new and existing tests passed
• I have added a changelog entry. Adding a new entry must be accomplished by running the scripts/new-change script and following the instructions. Commit the new file created by the script in .changes/next-release with your changes.
• My change is to implement 1.11 parity feature and I have updated LaunchChangelog

License

• I confirm that this pull request can be released under the Apache 2 license

[sonarcloud]

Quality Gate passed

Issues
3 New issues
0 Accepted issues

Measures
0 Security Hotspots
100.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud