Skip to content

Support Reactive Streams API for S3 protocol #198

Open
@migroskub

Description

@migroskub

Hey. I would like to suggest support for Project Reactor when integrating with AWS and Spring Integration. I suggest the ability to write a Flux to S3, consuming Mono and more. Here is a good example.

Activity

artembilan

artembilan commented on Dec 17, 2021

@artembilan
Member

<awssdk.version>2.10.27</awssdk.version>

We are aware about Reactive Streams API in the AWS SDK v2. Only the problem that we have to migrate into that version first and then we can see what we could do with the Reactive Streams support: #155.

I'm not closing this issue in favor of that, but rephrase it for S3 specifics, so we can come back to this when we do migration to AWS SDK v2.

changed the title [-]Project Reactor Support[/-] [+]Support Reactive Streams API for S3 protocol[/+] on Dec 17, 2021
migroskub

migroskub commented on Dec 17, 2021

@migroskub
Author

Ok. I didn't know about the SDK v2 capabilities. Sounds great. Can you tell about the timeline expectations for the SDK v2 migrations?

artembilan

artembilan commented on Dec 17, 2021

@artembilan
Member

Read the issue I've pointed in my answer.
And follow links to other related issues.
I cannot judge from here until we got some answers in the Spring Cloud AWS project.

lynch19

lynch19 commented on Dec 18, 2021

@lynch19

+1

almogtavor

almogtavor commented on Feb 10, 2022

@almogtavor

Are there any classes that can still work with internal implementation that is based on V2? Or maybe the case is that all of the existing classes doesn't support Reactor?

artembilan

artembilan commented on Feb 11, 2022

@artembilan
Member

Technically even existing AWS SKD v1 is compatible with Reactive Stream - its has an async variants for AWS services, which we really use in this Spring Integration for AWS project. So, you are good to develop Reactive solution even right now.

almogtavor

almogtavor commented on Feb 11, 2022

@almogtavor

Can you refer to the classes that are relevant for the asynchronous operations?

jifeiiii

jifeiiii commented on Mar 23, 2022

@jifeiiii

@artembilan So the outbound channel supports reactor or not? The docs (that doesn't appear in the website) doesn't mention this. Why?

jifeiiii

jifeiiii commented on Mar 23, 2022

@jifeiiii

I'd really like to see an example of usage of the OutboundGateway for S3 in the JavaDSL

artembilan

artembilan commented on Mar 23, 2022

@artembilan
Member

It would be great to determine in your opinion what is compatibility with Reactor or not.
As I said before: the AWS API in most cases is async, so it can be adapted to Reactor Flux and Mono.
There is no such a docs just because these component does not expose reactive API directly.
When they will, it would be reflected in docs.
However I treat any async API as compatible with reactive streams.
More over even blocking one could be shifted to the specific reactive scheduler do not block the whole stream, but wait for the result asynchronously.

The S3MessageHandler can be configured as a bean and used in the .handle() of Java DSL.
The produceReply option of it has to be set to true to make it working as a gateway.
It does return a com.amazonaws.services.s3.transfer.Transfer as a result of the publishing to S3.
You can adapter that object to reactive streams (Reactor Mono) via its public void addProgressListener(ProgressListener listener); hook.
But this already out of Spring Integration scope.

The docs is mentioning how to use an arbitrary channel adapter in Java DSL: https://docs.spring.io/spring-integration/docs/current/reference/html/dsl.html#java-dsl-protocol-adapters

jifeiiii

jifeiiii commented on Mar 28, 2022

@jifeiiii

@artembilan After looking at the code I see 2 things that makes me wonder if its really async and non blocking (e.g. safe to use with reactor).

  1. There is no use of S3AsyncClient
  2. There is usage of InputStream when passing byte[] to the upload command. Source - https://stackoverflow.com/questions/67476133/upload-a-inputstream-to-aws-s3-asynchronously-non-blocking-using-aws-sdk-for-j

3 remaining items

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

      Development

      No branches or pull requests

        Participants

        @artembilan@leoandrea7@almogtavor@lynch19@migroskub

        Issue actions

          Support Reactive Streams API for S3 protocol · Issue #198 · spring-projects/spring-integration-aws