Add `SnsMessageConverter` and `NotificationMessageArgumentResolver` to allow reading SNS messages on SQS #898

msosa · 2023-09-26T21:04:34Z

📢 Type of change

Bugfix
New feature
Enhancement
Refactoring

📜 Description

#887

💡 Motivation and Context

It allows for reading an SNS Message on an SQS Queue by using @NotficationMessage

💚 How did you test it?

Manually(will try to add tests when I have time)

📝 Checklist

I reviewed submitted code
I added tests to verify changes
I updated reference documentation to reflect the change
All tests passing
No breaking changes

🔮 Next steps

@tomazfernandes This is more of a draft for now, this works locally for me but was unsure where these files truly belong. I have put then in the sns project for now, since they use @NotificationMessage which lives there, and didn't want SQS project to have a dependency on the SNS one.

Let me know if you agree, or we should create a new annotation to be used.

Also in the previous implementation of NotificationRequestConverter it added MessageAttributes to the message. For my use-case I was able to remove that completely and things worked fine. Let me know if that is actually needed, but I did see SqsHeaderMapper has a lot of the same logic for what was being done for MessageAttributes

tomazfernandes · 2023-10-04T02:22:05Z

Hi @msosa, thanks for the PR.

I'm leaning towards having a separate annotation, since the modules are now separated, and I believe it would make sense for a project to consume an SNS message from SQS but not need anything from the SNS module itself.

We'd probably want to have a different name for the annotation so that it's not confusing for the users.

@MatejNedic, do you have any thoughts on this? Anything I might be missing regarding the integration between the modules?

Thanks

msosa · 2023-10-08T22:28:24Z

Makes sense, I have moved all the code into SQS and created a new annotation called SnsNotificationMessage(open to alternative names).

tomazfernandes

Hey @msosa, thanks, it's looking good.

Added a few comments, let me know if they make sense to you.

It would also be great if you can include an integration test so we can see it in action.

Thanks

.../main/java/io/awspring/cloud/sqs/annotation/AbstractListenerAnnotationBeanPostProcessor.java

...-sqs/src/main/java/io/awspring/cloud/sqs/support/converter/NotificationRequestConverter.java

msosa

I am now in the process of making the integration test. I have the test hitting the new code, however its failing due to the message I am providing.

I will fix that and complete the test in another commit later on

...-sqs/src/main/java/io/awspring/cloud/sqs/support/converter/NotificationRequestConverter.java

...qs/src/test/java/io/awspring/cloud/sqs/integration/SqsMessageConversionIntegrationTests.java

jerchende · 2023-10-17T06:59:45Z

Hi,
I just saw your merge request. I am currently working around this missing feature, since we want to update to spring boot 3. And I wonder if the MessageAttributes from the SNS notification don't need to be mapped into MessageHeader as well?
Best regards,
Johannes

gustavomonarin · 2024-02-20T10:29:39Z

Hello @msosa ,

Do you have plans to further work on this MR?

I have implemented something internally very similarly and would love to drop this code in favor of the opensource implementation.

The main difference was related to the MessageConverter. In our case we do use CloudEvents and the current MessageConverter is not able to handle well subtypes likes CloudEvent<MyEvent>. It is a super tiny change but that enables a lot. I would be more than happy to create a mr to your branch if you wish.

msosa · 2024-02-26T05:20:21Z

HI @gustavomonarin,

Yes I do plan on working on this MR, was just waiting for review(I just realized I may not have requested said review)

But yeah am open to checking out and change you mentioned and bringing into this MR

tomazfernandes

Looking good, left a few comments.

.../main/java/io/awspring/cloud/sqs/annotation/AbstractListenerAnnotationBeanPostProcessor.java

...-sqs/src/main/java/io/awspring/cloud/sqs/support/converter/NotificationRequestConverter.java

tomazfernandes · 2024-02-27T00:20:45Z

...-sqs/src/main/java/io/awspring/cloud/sqs/support/converter/NotificationRequestConverter.java

+	/**
+	 * Notification request wrapper.
+	 */
+	public static class NotificationRequest {


Maybe use a record instead?

tomazfernandes · 2024-02-27T00:21:39Z

...rc/main/java/io/awspring/cloud/sqs/support/resolver/NotificationMessageArgumentResolver.java

+	@Override
+	public Object resolveArgument(MethodParameter par, Message<?> msg) throws Exception {
+		Object object = this.converter.fromMessage(msg, par.getParameterType());
+		NotificationRequestConverter.NotificationRequest nr = (NotificationRequestConverter.NotificationRequest) object;


Lets use Assert.isInstance() here to throw a pretty message in case the instance doesn't match the expected class?

...qs/src/test/java/io/awspring/cloud/sqs/integration/SqsMessageConversionIntegrationTests.java

tomazfernandes · 2024-02-27T00:27:45Z

Hey @msosa, really sorry for the delay - I thought I was the one waiting for your reply. Left a few comments on the PR, please let me know if they make sense to you.

@gustavomonarin thanks for showing up. Can you share a bit more about what's your proposed change?

Thanks.

…solver` to allow reading SNS messages on SQS

msosa

No problem @tomazfernandes, I had it in draft so it was a bit confusing.

Also in regards to @jerchende's comment, I have implemented it locally similar to how it was done previously. But I had to make this method public static. Would it make sense to move NumberParser to its own class, so we can use the getNumberValue?

...s/src/main/java/io/awspring/cloud/sqs/annotation/SqsListenerAnnotationBeanPostProcessor.java

tomazfernandes · 2024-03-09T18:27:42Z

current MessageConverter is not able to handle well subtypes likes CloudEvent<MyEvent>

@gustavomonarin, can you elaborate on this please?

In order to easily support generic types, this PR makes the SnsMessageConverter a standard spring SmartMessageConverters. Smart Message Converter is deeply used within spring in a variety of different ways for compatiblity. The current main converter, the CompositeConverter is already a SmartConverter. This change branch contains the minimal and less intrusive changes, especially no touching open public abstract classes.

tomazfernandes · 2024-03-12T02:27:32Z

Hi @msosa and @tomazfernandes i mad a small change in the test to cover the case i mentioned.

Also provided two implementation solutions, a small one in the spring way of compatibility and a second one with a bit more ofbreaking change on the bean post processor.

The PR has not much deatils in the description but the two commit messages goes in the details, so i would suggest to review the commits individually.

@gustavomonarin I don't see your PR with these commits, where can I find it?

tomazfernandes · 2024-03-12T02:30:02Z

@msosa we're pretty much there!

The only thing missing is the documentation, let me know if you need any help with that.

Thanks!

gustavomonarin · 2024-03-12T09:42:51Z

Hi @msosa and @tomazfernandes i mad a small change in the test to cover the case i mentioned.

Also provided two implementation solutions, a small one in the spring way of compatibility and a second one with a bit more ofbreaking change on the bean post processor.

The PR has not much deatils in the description but the two commit messages goes in the details, so i would suggest to review the commits individually.

@gustavomonarin I don't see your PR with these commits, where can I find it?

It is on the @msosa original repo, as is based on his changes.

… `ObjectMapper` in `createAdditionalArgumentResolvers`

msosa · 2024-03-13T00:04:04Z

@tomazfernandes Awesome, appreciate all the feedback!

I gave the documentation a try, let me know what you think.

docs/src/main/asciidoc/sqs.adoc

tomazfernandes · 2024-03-13T00:30:52Z

@gustavomonarin, I looked at your suggestion to switch from implementing the MessageConverter interface to implementing the SmartMessageConverter interface which allows type hints and it makes sense.

I prefer the solution in this PR as it allows more flexibility for the user and is not a breaking change.

Now, not sure how to coordinate this - we do have an eminent version release coming up and it'd be great to have this feature in it.

@msosa, how would you feel about incorporating @gustavomonarin's changes into your PR?

…tification-message Sqs integration with sns notification message

msosa · 2024-03-13T01:26:31Z

@tomazfernandes of course, I have merged your preferred solution of @gustavomonarin into this PR.

Also I have updated the the documentation section to talk about SNS Messages

tomazfernandes

Almost there @msosa, just one comment on the documentation

docs/src/main/asciidoc/sqs.adoc

msosa · 2024-03-13T02:51:33Z

Removed the Note @tomazfernandes, let me know if there's anything else

tomazfernandes

Looking great @msosa!

Just a couple of finishing touches.

Please add @since 3.1.1 to the javadocs
Please add @gustavomonarin's name under yours in the relevant files
Please add "Since 3.1.1, when receiving SNS messages..."

That, and we're good to go!

Thanks!

msosa · 2024-03-14T02:32:26Z

@tomazfernandes Done! I just used @gustavomonarin username as I wasn't sure if exact first/last name.

tomazfernandes · 2024-03-14T02:42:06Z

@tomazfernandes Done! I just used @gustavomonarin username as I wasn't sure if exact first/last name.

Thanks @msosa!

@gustavomonarin, please let us know if you want us to name you in any other way.

tomazfernandes

Great work @msosa! And thanks @gustavomonarin for the improvement!

Looking forward for more!

coco1979ka · 2024-03-27T13:33:17Z

Looks great for Single Message processing. What is your recommendation to receive a List of SNSNotificationMessages for batch processing?

EstelaGarcia · 2024-04-03T11:48:25Z

We are also looking for a way to receive a List of SnsNotificationMessages and struggling with this. Any suggestions?

msosa requested review from maciejwalkowiak, MatejNedic and tomazfernandes as code owners September 26, 2023 21:04

github-actions bot added the component: sns SNS integration related issue label Sep 26, 2023

msosa marked this pull request as draft September 26, 2023 21:04

msosa force-pushed the sqs-integration-with-sns-notification-message branch from 3bfb74d to 60b1e3d Compare October 8, 2023 22:23

github-actions bot added the component: sqs SQS integration related issue label Oct 8, 2023

msosa force-pushed the sqs-integration-with-sns-notification-message branch from 60b1e3d to 3c81c9e Compare October 8, 2023 22:24

msosa force-pushed the sqs-integration-with-sns-notification-message branch 2 times, most recently from 00ce0a0 to d61525e Compare October 9, 2023 01:52

tomazfernandes requested changes Oct 12, 2023

View reviewed changes

msosa commented Oct 13, 2023

View reviewed changes

...-sqs/src/main/java/io/awspring/cloud/sqs/support/converter/NotificationRequestConverter.java Outdated Show resolved Hide resolved

...-sqs/src/main/java/io/awspring/cloud/sqs/support/converter/NotificationRequestConverter.java Outdated Show resolved Hide resolved

msosa commented Oct 16, 2023

View reviewed changes

...qs/src/test/java/io/awspring/cloud/sqs/integration/SqsMessageConversionIntegrationTests.java Show resolved Hide resolved

tomazfernandes mentioned this pull request Nov 23, 2023

@SqsListener Exception while handling pojo class #967

Closed

msosa marked this pull request as ready for review February 26, 2024 05:19

tomazfernandes reviewed Feb 27, 2024

View reviewed changes

msosa added 4 commits February 26, 2024 21:41

Add NotificationRequestConverter and `NotificationMessageArgumentRe…

630a41a

…solver` to allow reading SNS messages on SQS

updating code with suggestions

43744ef

adding integration test for NotificationMessageArgumentResolver

efdbc4d

review fixes

1f4c139

msosa force-pushed the sqs-integration-with-sns-notification-message branch from 401d7d0 to 1f4c139 Compare February 27, 2024 02:41

msosa commented Feb 27, 2024

View reviewed changes

...s/src/main/java/io/awspring/cloud/sqs/annotation/SqsListenerAnnotationBeanPostProcessor.java Outdated Show resolved Hide resolved

msosa requested a review from tomazfernandes February 27, 2024 02:51

gustavomonarin added 2 commits March 11, 2024 11:12

failing tests that deserialize object to hashmap.

73e8d49

Add documentation for SnsNotificationMessage and add null check for…

6b8398e

… `ObjectMapper` in `createAdditionalArgumentResolvers`

github-actions bot added the type: documentation Documentation or Samples related issue label Mar 13, 2024

msosa requested a review from tomazfernandes March 13, 2024 00:04

tomazfernandes reviewed Mar 13, 2024

View reviewed changes

docs/src/main/asciidoc/sqs.adoc Outdated Show resolved Hide resolved

msosa added 2 commits March 12, 2024 21:23

Update documentation to describe SNS Messages through SQS

5241e59

Merge pull request #1 from gustavomonarin/sqs-integration-with-sns-no…

a2986ed

…tification-message Sqs integration with sns notification message

msosa requested a review from tomazfernandes March 13, 2024 01:26

msosa changed the title ~~Add NotificationRequestConverter and NotificationMessageArgumentResolver to allow reading SNS messages on SQS~~ Add SnsMessageConverter and NotificationMessageArgumentResolver to allow reading SNS messages on SQS Mar 13, 2024

tomazfernandes reviewed Mar 13, 2024

View reviewed changes

docs/src/main/asciidoc/sqs.adoc Outdated Show resolved Hide resolved

Remove unneeded note

9a67895

msosa requested a review from tomazfernandes March 13, 2024 02:51

tomazfernandes reviewed Mar 14, 2024

View reviewed changes

Updating Javadocs

5970eda

msosa requested a review from tomazfernandes March 14, 2024 02:32

tomazfernandes approved these changes Mar 15, 2024

View reviewed changes

tomazfernandes merged commit 6ef09c0 into awspring:main Mar 15, 2024
4 checks passed

EstelaGarcia mentioned this pull request Apr 3, 2024

@SQSListener method: @SnsNotificationMessage parameter doesn't work in batching method signatures #1129

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add `SnsMessageConverter` and `NotificationMessageArgumentResolver` to allow reading SNS messages on SQS #898

Add `SnsMessageConverter` and `NotificationMessageArgumentResolver` to allow reading SNS messages on SQS #898

msosa commented Sep 26, 2023

tomazfernandes commented Oct 4, 2023

msosa commented Oct 8, 2023

tomazfernandes left a comment

msosa left a comment •

edited

Loading

jerchende commented Oct 17, 2023

gustavomonarin commented Feb 20, 2024 •

edited

Loading

msosa commented Feb 26, 2024

tomazfernandes left a comment

tomazfernandes Feb 27, 2024

tomazfernandes Feb 27, 2024

tomazfernandes commented Feb 27, 2024

msosa left a comment

tomazfernandes commented Mar 9, 2024 •

edited

Loading

tomazfernandes commented Mar 12, 2024

tomazfernandes commented Mar 12, 2024

gustavomonarin commented Mar 12, 2024 •

edited

Loading

msosa commented Mar 13, 2024

tomazfernandes commented Mar 13, 2024

msosa commented Mar 13, 2024

tomazfernandes left a comment

msosa commented Mar 13, 2024

tomazfernandes left a comment

msosa commented Mar 14, 2024

tomazfernandes commented Mar 14, 2024

tomazfernandes left a comment

coco1979ka commented Mar 27, 2024

EstelaGarcia commented Apr 3, 2024

Add SnsMessageConverter and NotificationMessageArgumentResolver to allow reading SNS messages on SQS #898

Add SnsMessageConverter and NotificationMessageArgumentResolver to allow reading SNS messages on SQS #898

Conversation

msosa commented Sep 26, 2023

📢 Type of change

📜 Description

💡 Motivation and Context

💚 How did you test it?

📝 Checklist

🔮 Next steps

tomazfernandes commented Oct 4, 2023

msosa commented Oct 8, 2023

tomazfernandes left a comment

Choose a reason for hiding this comment

msosa left a comment • edited Loading

Choose a reason for hiding this comment

jerchende commented Oct 17, 2023

gustavomonarin commented Feb 20, 2024 • edited Loading

msosa commented Feb 26, 2024

tomazfernandes left a comment

Choose a reason for hiding this comment

tomazfernandes Feb 27, 2024

Choose a reason for hiding this comment

tomazfernandes Feb 27, 2024

Choose a reason for hiding this comment

tomazfernandes commented Feb 27, 2024

msosa left a comment

Choose a reason for hiding this comment

tomazfernandes commented Mar 9, 2024 • edited Loading

tomazfernandes commented Mar 12, 2024

tomazfernandes commented Mar 12, 2024

gustavomonarin commented Mar 12, 2024 • edited Loading

msosa commented Mar 13, 2024

tomazfernandes commented Mar 13, 2024

msosa commented Mar 13, 2024

tomazfernandes left a comment

Choose a reason for hiding this comment

msosa commented Mar 13, 2024

tomazfernandes left a comment

Choose a reason for hiding this comment

msosa commented Mar 14, 2024

tomazfernandes commented Mar 14, 2024

tomazfernandes left a comment

Choose a reason for hiding this comment

coco1979ka commented Mar 27, 2024

EstelaGarcia commented Apr 3, 2024

Add `SnsMessageConverter` and `NotificationMessageArgumentResolver` to allow reading SNS messages on SQS #898

Add `SnsMessageConverter` and `NotificationMessageArgumentResolver` to allow reading SNS messages on SQS #898

msosa left a comment •

edited

Loading

gustavomonarin commented Feb 20, 2024 •

edited

Loading

tomazfernandes commented Mar 9, 2024 •

edited

Loading

gustavomonarin commented Mar 12, 2024 •

edited

Loading