@SqsListener not working after upgrading to Spring Boot 3.1.3

[julpalma]

Type: Bug

Component:

SQS

Describe the bug
Please provide details of the problem, including the version of Spring Cloud that you
are using.

After upgrading to Spring Boot 3.1.3, @SqsListener is not working anymore. My service is able to send a message to SQS queue but is not receiving the message. I have been trying to fix for more than 3 weeks now, but cant seem to figure this out.
Can someone please help me figure out what is missing?? Thank you!!!!

Note that the code below is for local testing. With Spring Boot 2.x it works fine.

This is my Configuration code, in the Configuration class:

@Bean
    public SqsAsyncClient sqsAsyncClient(){
        return SqsAsyncClient.builder().endpointOverride(URI.create("http://localhost:9324"))
                .region(Region.US_EAST_1)
                .credentialsProvider(StaticCredentialsProvider.create(new AwsCredentials() {
                    @Override
                    public String accessKeyId() {
                        return "Local";
                    }

                    @Override
                    public String secretAccessKey() {
                        return "local";
                    }
                }))
                .build();
    }

    @Bean
    public SqsTemplate sqsTemplateManualContainerInstantiation(SqsAsyncClient sqsAsyncClient) {
        return SqsTemplate.builder()
                .sqsAsyncClient(sqsAsyncClient)
                .build();
    }

    @Bean
    SqsMessageListenerContainerFactory defaultSqsListenerContainerFactory(SqsAsyncClient sqsAsyncClient) {
        return SqsMessageListenerContainerFactory
                .builder()
                .configure(options -> options
                        .pollTimeout(Duration.ofSeconds(10)))
                .sqsAsyncClient(sqsAsyncClient)
                .messageListener((message) -> log.info("Received SQS message {}", message))
                .build();
    }

This is the sendMessage code in the Service class.

public void sendMessage(Request request, User user) {
        try {
            MyServiceRequest payload = new MyServiceRequest(user, request.getEvent(), TrackingUtil.getTrackingId(), Instant.now().getEpochSecond());
            sqsTemplate.send("queueUrl", buildMessageRequest(payload));
        } catch (Exception e) {
            log.error("Failed to send message to SQS for user {}, cause by {}", user, e.getMessage());
        }
    }

private SendMessageRequest buildMessageRequest(MyServiceRequest payload) {
        MessageAttributeValue contentTypeJson = new MessageAttributeValue()
                .withDataType("String")
                .withStringValue("application/json");
        return new SendMessageRequest("queueUrl", toSqsFormat(payload))
                .addMessageAttributesEntry("contentType", contentTypeJson);
    }

    private String toSqsFormat(MyServiceRequest payload) {
        try {
            return objectMapper.writeValueAsString(payload);
        } catch (IOException e) {
            throw new IllegalArgumentException("Failed to convert payload " + message + " to JSON", e);
        }
    }

This is the listener code:

@SqsListener(value = "queueUrl", deletionPolicy = SqsMessageDeletionPolicy.ON_SUCCESS)
    public void onMessageReceived(MyServiceRequest message) {
        log.info("Received message from SQS");
    }

These are the AWS dependencies in the pom file:

   <dependency>
        <groupId>io.awspring.cloud</groupId>
        <artifactId>spring-cloud-aws-messaging</artifactId>
        <version>2.4.4</version>
    </dependency>

   <dependency>
        <groupId>io.awspring.cloud</groupId>
        <artifactId>spring-cloud-aws-starter-sqs</artifactId>
        <version>3.0.2</version>
    </dependency>

When i run the application with debug mode, i see this:

SqsAutoConfiguration matched:
- @ConditionalOnClass found required classes 'software.amazon.awssdk.services.sqs.SqsAsyncClient', 'io.awspring.cloud.sqs.config.SqsBootstrapConfiguration' (OnClassCondition)
- @ConditionalOnProperty (spring.cloud.aws.sqs.enabled=true) matched (OnPropertyCondition)

SqsAutoConfiguration#defaultSqsListenerContainerFactory:
Did not match:
- @ConditionalOnMissingBean (types: io.awspring.cloud.sqs.config.SqsMessageListenerContainerFactory; SearchStrategy: all) found beans of type 'io.awspring.cloud.sqs.config.SqsMessageListenerContainerFactory' defaultSqsListenerContainerFactory (OnBeanCondition)

SqsAutoConfiguration#sqsAsyncClient:
Did not match:
- @ConditionalOnMissingBean (types: software.amazon.awssdk.services.sqs.SqsAsyncClient; SearchStrategy: all) found beans of type 'software.amazon.awssdk.services.sqs.SqsAsyncClient' sqsAsyncClient (OnBeanCondition)

SqsAutoConfiguration#sqsTemplate:
Did not match:
- @ConditionalOnMissingBean (types: io.awspring.cloud.sqs.operations.SqsTemplate; SearchStrategy: all) found beans of type 'io.awspring.cloud.sqs.operations.SqsTemplate' sqsTemplateManualContainerInstantiation (OnBeanCondition)

In addition to that, when i check the QueueUrl, i see this:

Sample
If possible, please provide a test case or sample application that reproduces
the problem. This makes it much easier for us to diagnose the problem and to verify that
we have fixed it.

[maciejwalkowiak]

You're mixing Spring Cloud AWS 2.x with 3.x. If you're using Spring Boot 3, you must use Spring Cloud AWS 3.x. I am using SQS integration with Spring Boot 3.1.4 so definitely unless there is some unusual config, it does work as expected.

Please try adjusting dependencies and let us know if it helped.

[julpalma]

Thank you so much for the reply @maciejwalkowiak. Understood.
I have updated my dependencies to:

io.awspring.cloud
spring-cloud-aws-starter-sqs
3.0.2


org.springframework
spring-messaging


software.amazon.awssdk
sqs
2.20.63


com.amazonaws
aws-java-sdk-sqs


io.awspring.cloud
spring-cloud-aws-core
3.0.2


org.springframework.boot
spring-boot-starter-web

I have updated my code:

Configuration:

@Bean
    public SqsTemplate sqsTemplateManualContainerInstantiation() {
        return SqsTemplate.builder()
                .sqsAsyncClient(sqsAsyncClient())
                .build();
    }

    @Bean
    public SqsMessageListenerContainerFactory<Object> defaultSqsListenerContainerFactory() {
        return SqsMessageListenerContainerFactory
                .builder()
                .sqsAsyncClient(sqsAsyncClient())
                .build();
    }

@Bean
    public SqsAsyncClient sqsAsyncClient() {
        return SqsAsyncClient.builder().endpointOverride(URI.create("http://localhost:9324/000000000000/standard"))
                .region(Region.US_EAST_1)
                .credentialsProvider(StaticCredentialsProvider.create(new AwsCredentials() {
                    @Override
                    public String accessKeyId() {
                        return "LOCAL";
                    }

                    @Override
                    public String secretAccessKey() {
                        return "LOCAL";
                    }
                }))
                .build();
    }

Code to sendMessage:

public void sendMessage(Request request, User user) {
try {
MyServiceRequest payload = new MyServiceRequest(user, request.getEvent(), TrackingUtil.getTrackingId(), Instant.now().getEpochSecond());
sqsTemplate.send("queueUrl", toSqsFormat(payload));
} catch (Exception e) {
log.error("Failed to send message to SQS for user {}, cause by {}", user, e.getMessage());
}

private String toSqsFormat(MyServiceRequest message) {
        try {
            return objectMapper.writeValueAsString(message);
        } catch (IOException e) {
            throw new IllegalArgumentException("Failed to convert MyServiceRequest " + message + " to JSON", e);
        }
    }

Code to receive message:

@SqsListener(value = "queueUrl")
public void onMessageReceived(MyServiceRequest message) {
log.info("Received message from SQS");
}

I have imported this SqsBootstrapConfiguration.class as well in the Configuration class.

I am able to send the message to the queue but i have this error:

org.springframework.messaging.converter.MessageConversionException: Could not read JSON: Cannot construct instance of com.amazonaws.event.ProgressListener (no Creators, like default constructor, exist): abstract types either need to be mapped to concrete types, have custom deserializer, or contain additional type information
at [Source: (String)"{"requestClientOptions":{"readLimit":131073,"skipAppendUriPath":false},"requestMetricCollector":null,"customRequestHeaders":null,"customQueryParameters":null,"cloneSource":null,"sdkRequestTimeout":null,"sdkClientExecutionTimeout":null,"queueUrl":"http://localhost:9324/000000000000/standard","messageBody":"{"userId":"00000000-0000-0000-0000-000000000000","eventType":"call","trackingId":"NA_66359720-b056-42ea-b2bd-fc33f9416339","sentAt":1697828192}","delaySeconds":null,"messageAttr"[truncated 367 chars]; line: 1, column: 810] (through reference chain: com.amazonaws.services.sqs.model.SendMessageRequest["generalProgressListener"])
at org.springframework.messaging.converter.MappingJackson2MessageConverter.convertFromInternal(MappingJackson2MessageConverter.java:235)
at org.springframework.messaging.converter.AbstractMessageConverter.fromMessage(AbstractMessageConverter.java:185)
at org.springframework.messaging.converter.AbstractMessageConverter.fromMessage(AbstractMessageConverter.java:176)
at org.springframework.messaging.converter.CompositeMessageConverter.fromMessage(CompositeMessageConverter.java:57)
at io.awspring.cloud.sqs.support.converter.AbstractMessagingMessageConverter.convertPayload(AbstractMessagingMessageConverter.java:172)
at io.awspring.cloud.sqs.support.converter.AbstractMessagingMessageConverter.toMessagingMessage(AbstractMessagingMessageConverter.java:153)

Anything else is incorrect or anything else i am missing??

Thank you so much!!!

Juliana

[maciejwalkowiak]

I find it difficult to go through your code samples, formatting could be one of the reasons. Also, com.amazonaws.event.ProgressListener comes from AWS SDK v1.

Are you able to publish a complete project that reproduces your issue? Something I can checkout and run myself?

[julpalma]

Thank you for the reply @maciejwalkowiak. Just updating here with my latest status.

I had this dependency on my pom file:
com.amazonaws
aws-java-sdk-bom
1.12.395
pom
import

I removed and added:
software.amazon.awssdk
bom
2.20.63
pom
import

I see that the Listener container has started successfully now:
I see this in the logs:
"Container io.awsspring.cloud.sqs.sqsListenerEndpointContainer#0 started"

I dont see that exception anymore.

However, i have this one exception for the token:

11:13:52.722 ERROR --- [nc-response-0-0] i.a.c.s.o.AbstractMessagingTemplate : Error sending message 90593638-c193-7a72-230c-85614f6c7d6a to endpoint http://localhost:9324/000000000000/standard

io.awspring.cloud.sqs.operations.MessagingOperationFailedException: Message send operation failed for message 90593638-c193-7a72-230c-85614f6c7d6a to endpoint http://localhost:9324/000000000000/standard
at io.awspring.cloud.sqs.operations.AbstractMessagingTemplate.lambda$sendAsync$11(AbstractMessagingTemplate.java:285)
at java.base/java.util.concurrent.CompletableFuture$UniComposeExceptionally.tryFire(CompletableFuture.java:1040)
at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)

.....

Caused by: software.amazon.awssdk.services.sqs.model.SqsException: The security token included in the request is invalid. (Service: Sqs, Status Code: 403, Request ID: 4c1d683e-87b8-5c5e-817e-358db548edfe)
at software.amazon.awssdk.services.sqs.model.SqsException$BuilderImpl.build(SqsException.java:104)

This is my sqsClient:

public SqsAsyncClient sqsAsyncClient() {
        return SqsAsyncClient.builder()
                .region(Region.US_EAST_1)
                .credentialsProvider(StaticCredentialsProvider.create(new AwsCredentials() {
                    @Override
                    public String accessKeyId() {
                        return LOCAL;
                    }

                    @Override
                    public String secretAccessKey() {
                        return LOCAL;
                    }
                }))
                .build();
    }

[julpalma]

And i see this error:

The Canonical String for this request should have been
'POST
/

amz-sdk-invocation-id:5c407020-847c-c2a7-c4bc-86917bfee56e
amz-sdk-request:attempt=1; max=4
content-length:194
content-type:application/x-www-form-urlencoded; charset=utf-8
host:sqs.us-east-1.amazonaws.com
x-amz-date:20231025T181352Z

amz-sdk-invocation-id;amz-sdk-request;content-length;content-type;host;x-amz-date
522127270de60df776f808abc2797e1c1edb0b79baff2350e4b6f62651c3e3d8'

The String-to-Sign should have been
'AWS4-HMAC-SHA256
20231025T181352Z
20231025/us-east-1/sqs/aws4_request
c867198790efaa4356d16af8d2bae63bf886f61f84c726d5f94ab42c4ff80b82'
(Service: Sqs, Status Code: 403, Request ID: c25fc9d1-af12-55e0-95db-9462f232a84e)
at software.amazon.awssdk.services.sqs.model.SqsException$BuilderImpl.build(SqsException.java:104)

[julpalma]

@maciejwalkowiak we found the issue.

The host was set to the real server, when we overwrite, it works fine. Thank you for your help. You can close this issue.

[maciejwalkowiak]

Uff super happy you found it! I am closing it then!

[tmdgusya]

@maciejwalkowiak Thank you for kindness answering for us. Do you happen to have any documents for migrating 2.X to 3.X version related to aws-messaging? because I want to know what classes that I should use when migrate current class to higher version (migrated) classes.