S3Template fails to upload file when using client side encryption

[gbaso]

Type: Bug

Component:
S3

Describe the bug
s3Template.upload(bucket, key, inputStream) fail when a bean of type S3EncryptionClient is defined, with S3Exception: The Content-MD5 you specified did not match what we received.

Analyzing the HTTP exchange with wireshark shows that S3Template adds the header Content-MD5 with the checksum of the plaintext file. However, S3 computes the checksum of the encrypted payload, causing a mismatch.

S3Client sidesteps this issue by not providing a Content-MD5 header in the request, so s3Client.putObject(put -> put.bucket(bucket).key(key), inputFile) works as expected.

Sample
Example application with instructions and a comparison of HTTP exchanges: https://github.com/gbaso/spring-cloud-aws-s3-encryption

[MatejNedic]

Hey @gbaso tnx on reporting!

Are you interested in contributing?

[gbaso]

I'm not sure what the best approach would be. Avoid computing MD5 checksum when S3Client instanceof S3EncryptionClient? Feels a bit brittle. Disable it altogether, given that it's optional and the S3Client doesn't set it? Looking at the documentation, x-amz-checksum-* headers seem preferred for object integrity, then again by default the value of x-amz-checksum-sha256 set by the S3Client (both encrypted and not encrypted) is UNSIGNED-PAYLOAD...

The change itself seems to be localized to InMemoryBufferingS3OutputStream and AbstractTempFileS3OutputStream, so it should not be a problem.

[MatejNedic]

Hmmm, tbh I would remove checksum fully if encryptionClient is present. I wouldn't even calculate it not sure if it makes sense.

What do you think?

[gbaso]

I'm not sure it's even possible to instruct the encryption client to compute a checksum after the encryption. Encrypting the file ourself for the checksum is not doable since the data key and possibly any initialization parameters would be different from the actual encrypted file to upload.

Would you still compute the MD5 checksum when using the plain S3Client?

[gbaso]

Moreover, amazon-s3-encryption-client-java is not a dependency of spring-cloud-aws-s3, so any check on the encryption client would require adding it as an optional dependency.