S3 Get Bucket Location does not consistently fail depending on client region

[rpless]

I'm running into an issue with the S3 client making Get Bucket Location failing on calls for buckets in other regions, but only when the client is configured for us-east-1. In the example below I make an S3 client with the region of us-east-1 and try to make a Get Bucket Location call to a bucket that is in us-west-2 (although I have confirmed this the same behavior for buckets in us-west-1, us-east-2, eu-west-1, eu-west-2). This fails with with an AuthorizationHeaderMalformed error. Issue #1142 seems to suggest that this is the expected behavior.

However, also in the example I make a client for region us-west-2 (I have also test this with clients for us-west-1, us-east-2, eu-west-1, eu-west-2) and make a Get Bucket Location to a bucket in us-east-1 and it correctly resolves the region. I can also make Get Bucket Location calls with this client to resolve the bucket location of any bucket in these regions. Based on #1142 this is not expected behavior.

Should I expect S3 clients configured for us-east-1 to not be able to make Get Bucket Location calls to other regions, but clients for other regions to be fine? Or should the behavior consistently be an error?
I've implemented one of the workarounds from #1142, but consistency in how this call works would be ideal.

Java Version: 1.8.0_102
AWS SDK Version: 1.11.211

public class S3Reproducer {
    public static void main(String[] args) {

        // I've also tested this with a STSAssumeRoleSessionCredentialsProvider 
        AWSCredentialsProvider credentials = new ClasspathPropertiesFileCredentialsProvider();

        String bucketInUsEast1 = "<name of bucket in us-east-1>";
        String bucketInUsWest2 = "<name of bucket in us-west-2>"; // or any of the above mentioned regions

        AmazonS3 usEast1Client = AmazonS3ClientBuilder
            .standard()
            .withRegion(Regions.US_EAST_1)
            .withCredentials(credentials)
            .build();

        AmazonS3 usWest2Client = AmazonS3ClientBuilder
            .standard()
            .withRegion(Regions.US_WEST_2)
            .withCredentials(credentials)
            .build();


        System.out.println("Result of querying a bucket in us-east-1 from us-west-2:");
        String usEast1BucketLocation = usWest2Client.getBucketLocation(bucketInUsEast1);
        System.out.println(usEast1BucketLocation);

        System.out.println("Result of querying a bucket in us-west-2 from us-east-1:");
        String usWest2BucketLocation = usEast1Client.getBucketLocation(bucketInUsWest2); // Fails  
    }
}

The output I see is:

Result of querying a bucket in us-east-1 from us-west-2:
US
Result of querying a bucket in us-west-2 from us-east-1:
Exception in thread "main" com.amazonaws.services.s3.model.AmazonS3Exception: The authorization header is malformed; the region 'us-east-1' is wrong; expecting 'us-west-2' (Service: Amazon S3; Status Code: 400; Error Code: AuthorizationHeaderMalformed; // rest of error elided

[fernomac]

What you’re seeing is expected, for better or worse. The underlying problem is that S3’s us-east-1 endpoint is a bit “special”. It’s simply s3.amazonaws.com, and when used with bucket-style addressing (the default), bucket.s3.amazonaws.com will resolve to whichever actual s3 region the bucket is in. This was handy when s3 used an authentication mechanism that was not region-aware, but these days you need to know what region you’re talking to to generate a valid signature.

This overlaps in interesting ways with GetBucketLocation, which is a “global” API — you can call it on any region you want irrespective of the bucket, so long as you sign the request appropriately for the region you send it to. When you configure the client for “us-east-1” but the request actually ends up in the “right” place because of DNS, the signature is rejected.

Amazon folks: I’d still love to see a proper s3.us-east-1.amazonaws.com endpoint without the extra semantics, and for the SDK to switch to use it when configured for us-east-1 (maybe in v2 if you want to be safe). Eh eh eh?

[rpless]

Thanks @fernomac that makes sense to me, its just frustrating to have that inconsistency with these "global" api calls.

This would be great to see in V2, but in the mean time maybe documentation could be added to region agnostic api calls to indicate that this behavior can happen.

[kiiadi]

We're working with S3 to try to normalize this; although as @fernomac points out the underlying issue is with the service. We're looking at ways to make the client implementations behave more consistently for v2 see related backlog item : aws/aws-sdk-java-v2#52