Make S3 "put" report on non-ASCII metadata characters more clearly

[timorantalaiho]

Describe the feature

A file upload to S3 failed with
software.amazon.awssdk.services.s3.model.S3Exception: The request signature we calculated does not match the signature you provided. Check your key and signing method.

After an investigation, we realised this was because there were non-ASCII characters in the metadata we provided.

When debugging, I found out that the CLI reports this a lot more clearly:

> aws s3 cp --metadata 'KeyName1=Ä is a bad character here' /tmp/lol s3://fnu8cjrfc3ju89rf389r3u89ru8989r3-test-bucket/zomg.dat
upload failed: ../../tmp/lol to s3://fnu8cjrfc3ju89rf389r3u89ru8989r3-test-bucket/zomg.dat Parameter validation failed:
Non ascii characters found in S3 metadata for key "KeyName1", value: "Ä is a bad character here".
S3 metadata can only contain ASCII characters.

Use Case

"The request signature we calculated does not match the signature you provided" is misleading, when in fact the problem is in bad characters in the metadata.

Error analysis would be quicker with a better error message.

Proposed Solution

Make the SDK S3 putObject validate and report metadata input similarly as the aws s3 CLI.

Other Information

I do understand that you do not want to automatically escape non-ASCII characters in any SDK (see e.g. aws/aws-sdk-js#1949 ), because it would be a breaking change.

But surely a better error message for the S3Exception would be a safe and useful change?

Acknowledgements

• I may be able to implement this feature request
• This feature might incur a breaking change

AWS Java SDK version used

2.21.14

JDK version used

openjdk version "17.0.9" 2023-10-17 LTS OpenJDK Runtime Environment Corretto-17.0.9.8.1 (build 17.0.9+8-LTS) OpenJDK 64-Bit Server VM Corretto-17.0.9.8.1 (build 17.0.9+8-LTS, mixed mode, sharing)

Operating System and version

Linux 5.15.0-88-generic aws/aws-sdk-java-v2#98~20.04.1-Ubuntu SMP Mon Oct 9 16:43:45 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

[debora-ito]

@timorantalaiho thank you for reaching out, and I'm sorry for the long silence.

As a general rule, the Java SDK does not validate attributes on the client side, because (1) by doing this we avoid forcing people to upgrade the SDK version in case the service decides to change the validation rules in the future; and (2) we can't maintain custom validation code for all aws services if it's not something modeled.

The Java SDK does automatically encode some attributes, like bucket name and object name, but not metadata. And you're right, we can't start enconding metadata now because it's a breaking change. The exception you see is the exception sent by the service.

Let us know if you have any more questions.

[timorantalaiho]

Thankyou!

Do you think it might be worthwhile to report this to S3 then, to see if they could provide a better exception in these cases?

[bhoradc]

Moving this issue to Shared SDK repo for better tracking. I will reach out to the S3 service team to check if the exception in this case can be made more informative.

[bhoradc]

V1470349306

[tim-finnigan]

Thanks for your patience. The S3 team is continuing to track this in their backlog. We cannot guarantee if or when service API feature requests like these would be implemented, but please refer to the blog or SDK changelog for updates going forward.

[github-actions]

This issue is now closed.

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.