aws sync broken since 2.1.29

[mibollma]

aws sync seems to be broken since 2.1.29.
Calling the command in it's basic form e.g. aws s3 sync . s3://bucket --endpoint-url http://localhost:9090 produces the error fatal error: 'Prefix'.
The same command works fine till version 2.1.28.

[afranken]

I just reproduced this locally.
In #251, which was released in 2.1.29, support for the "encodingType" parameter was added.

Unfortunately, this seems to have broken object listing, as the client expects several field values in the response to be encoded (Prefix for example). If the field value is empty, the client expects those fields to be dropped from the response.
The error is thrown because we always return the Prefix and CommonPrefix fields with empty values, which the client tries to decode without checking if the value is empty.

[afranken]

API for reference:
https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectsV2.html#API_ListObjectsV2_ResponseSyntax

[gerhalt]

Ran into exactly the same original URL-encoding issue and the new breakage described here.

[afranken]

I have a fix ready, just working on the tests right now - unfortunately the V1 and V2 APIs (or rather the client implementation of ListObjects vs ListObjectsV2) do different things when it comes to responses with encoding.

I'm wondering if we have to keep supporting the V1 APIs in the mock...

[mibollma]

I'm still using V1 currently. One reason is that some features were only available in V1 at least until recently, such as DynamoDB client-side encryption: aws/aws-sdk-java-v2#34

[afranken]

I did not mean the Java SDK V1, but the API:
https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjects.html
https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectsV2.html

It is possible to request both ListObjects as well as ListObjectsV2 with the V1 SDK:
V1
https://github.com/aws/aws-sdk-java/blob/master/aws-java-sdk-s3/src/main/java/com/amazonaws/services/s3/AmazonS3.java#L543
V2
https://github.com/aws/aws-sdk-java/blob/master/aws-java-sdk-s3/src/main/java/com/amazonaws/services/s3/AmazonS3.java#L552