Open
Description
Hey. I would like to suggest support for Project Reactor when integrating with AWS and Spring Integration. I suggest the ability to write a Flux
to S3, consuming Mono
and more. Here is a good example.
Hey. I would like to suggest support for Project Reactor when integrating with AWS and Spring Integration. I suggest the ability to write a Flux
to S3, consuming Mono
and more. Here is a good example.
Activity
artembilan commentedon Dec 17, 2021
We are aware about Reactive Streams API in the AWS SDK v2. Only the problem that we have to migrate into that version first and then we can see what we could do with the Reactive Streams support: #155.
I'm not closing this issue in favor of that, but rephrase it for S3 specifics, so we can come back to this when we do migration to AWS SDK v2.
[-]Project Reactor Support[/-][+]Support Reactive Streams API for S3 protocol[/+]migroskub commentedon Dec 17, 2021
Ok. I didn't know about the SDK v2 capabilities. Sounds great. Can you tell about the timeline expectations for the SDK v2 migrations?
artembilan commentedon Dec 17, 2021
Read the issue I've pointed in my answer.
And follow links to other related issues.
I cannot judge from here until we got some answers in the Spring Cloud AWS project.
lynch19 commentedon Dec 18, 2021
+1
almogtavor commentedon Feb 10, 2022
Are there any classes that can still work with internal implementation that is based on V2? Or maybe the case is that all of the existing classes doesn't support Reactor?
artembilan commentedon Feb 11, 2022
Technically even existing AWS SKD v1 is compatible with Reactive Stream - its has an async variants for AWS services, which we really use in this Spring Integration for AWS project. So, you are good to develop Reactive solution even right now.
almogtavor commentedon Feb 11, 2022
Can you refer to the classes that are relevant for the asynchronous operations?
artembilan commentedon Feb 11, 2022
See this one and all its extensions in the same package: https://github.com/spring-projects/spring-integration-aws/blob/main/src/main/java/org/springframework/integration/aws/outbound/AbstractAwsMessageHandler.java.
jifeiiii commentedon Mar 23, 2022
@artembilan So the outbound channel supports reactor or not? The docs (that doesn't appear in the website) doesn't mention this. Why?
jifeiiii commentedon Mar 23, 2022
I'd really like to see an example of usage of the OutboundGateway for S3 in the JavaDSL
artembilan commentedon Mar 23, 2022
It would be great to determine in your opinion what is compatibility with Reactor or not.
As I said before: the AWS API in most cases is async, so it can be adapted to Reactor
Flux
andMono
.There is no such a docs just because these component does not expose reactive API directly.
When they will, it would be reflected in docs.
However I treat any async API as compatible with reactive streams.
More over even blocking one could be shifted to the specific reactive scheduler do not block the whole stream, but wait for the result asynchronously.
The
S3MessageHandler
can be configured as a bean and used in the.handle()
of Java DSL.The
produceReply
option of it has to be set totrue
to make it working as a gateway.It does return a
com.amazonaws.services.s3.transfer.Transfer
as a result of the publishing to S3.You can adapter that object to reactive streams (Reactor
Mono
) via itspublic void addProgressListener(ProgressListener listener);
hook.But this already out of Spring Integration scope.
The docs is mentioning how to use an arbitrary channel adapter in Java DSL: https://docs.spring.io/spring-integration/docs/current/reference/html/dsl.html#java-dsl-protocol-adapters
jifeiiii commentedon Mar 28, 2022
@artembilan After looking at the code I see 2 things that makes me wonder if its really async and non blocking (e.g. safe to use with reactor).
S3AsyncClient
InputStream
when passingbyte[]
to theupload
command. Source - https://stackoverflow.com/questions/67476133/upload-a-inputstream-to-aws-s3-asynchronously-non-blocking-using-aws-sdk-for-j3 remaining items