Automatic Request Batching

[shorea]

Batch write operations are a common use case across many AWS services. There is opportunity to improve the developer experience in regards to these style operations. As mentioned in aws/aws-sdk-java#1162, one idea is to expose an interface that takes an iterable or stream of results and a batch size and automatically chunks that into a batch write request and makes multiple calls until the iterable/stream is completed. Another potential idea is to automatically retry failed items in batches using the retry policy of the client.

[shorea]

Per comments from @wogan in the gitter channel.

@shorea ideally the interface would expose the successful and failed methods so I could have generic code that can inspect a batch response and ensure it was entirely successful or not
also a lot of the BatchEntry classes are extremely similar so simple logging handlers on errors could be written generically

[tinexw]

Hi,

I just wanted to know if there are any news regarding the AmazonSQSBufferedAsyncClient? I'm currently working on upgrading Spring Cloud AWS to the new SDK and it relies on this class (s. spring-attic/spring-cloud-aws#267).

Thanks!

[zoewangg]

This feature is on our radar, but we don't have a release date at this moment. Please 👍🏽 on this issue, which will help us prioritize.

[radifalco]

Any progress on this? It's really the only thing keeping me on v1. I'm also fine with a well-supported project that does it outside of this repo.

[crzyjcky]

This ticket has been opened since 2017, what's holding it back?

[chris-afterpay]

Any update with this one please?

[tgracchus]

We migrated to v2 and we are really missing this feature.
Please, make sure to include in the new sdks

[GlitchedTam]

Please! We need this feature in our projects in the company too!

[jwang-r7]

Plenty of Spring 2.x applications that are using the SDK will be EOL on Nov 18 2023. This means we have to use Java 17 as a baseline. However, AWS SDK 1.x does not support Java 17. Apps that currently use the SQS buffered client will likely be forced to see their costs tenfold if they were to migrate due to the EOL. Can we add some urgency to this issue?

[adrian-skybaker]

It seems the idea of a generic request batching concept across the whole SDK is reasonably ambitious, given the 6 year period without progress. What about implementing a straight port of the 1.x SQS buffered client to 2.x in the meantime? ie, reopen #4576 and implement a like-for-like that uses the same approach and code but on the V2 SQS API.

[debora-ito]

This feature is high up in our priority list. We are fairly confident we'll release it in the next quarter - we are so confident that I'm sharing the timeline publicly so you all can hold us accountable later on. Right now, we are busy working on internal refactoring that's important for future features, plus some re:Invent work. Once those things are done we'll move on to the next items on the priority list.

As for implementing a generic request batching vs. porting the v1 SQS buffered client to v2, porting the client from v1 to v2 is not a quick task, it needs to be fully rewritten into the v2 architecture anyway. The plan is still to release a generic request batching.

[kunal15595]

we are nearing this quarter end, is this capability available in v2 SDK now?

[camerondurham]

@debora-ito Any updated timeline on releasing v2 async buffered SQS client?

[pjy1368]

@debora-ito Any updated timeline on releasing v2 async buffered SQS client?

[joviegas]

We have released #5580 and its available in aws sdk java v2 2.28.0 and higher

[github-actions]

This issue is now closed. Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.