Cross-Region S3 Client

[millems]

The 1.11.x S3 client automatically routed requests to the correct bucket region in a complicated manner. The 2.0.x client does not perform this automatic re-routing.

We should create a wrapper to pool S3 clients for different regions and automatically route requests to the region in which the bucket was created.

[manasshukla]

Is there an update on when this feature would be shipped out ?

[millems]

Sorry, no update.

It's not our #1 priority, because with access points and other arn-based ways of accessing buckets/objects plus the use-arn-region flag, the SDK will automatically route the request to the correct region without the overhead of an extra call to AWS to find out where the bucket actually is.

I understand there's still value in this feature request, since the caller may not have a full bucket ARN.

[byF]

If anyone thinks using ARNs like arn:aws:s3:::mybucket will work, they won't #2244

[pkgonan]

@millems Hi. Any plan of supporting this feature ? A lot of people think they need this feature from the perspective of customers using AWS S3 products.

[millems]

At some point, probably, but in the meantime you could try using access points which automatically support cross-region request routing.

[pkgonan]

@millems
Oh. That's good news. When are you expecting?

[millems]

@pkgonan Unfortunately we have no expected date to report at this time.

[pkgonan]

@millems Thanks for your reply. :)

[maciejwalkowiak]

FYI we are working on adding cross-region S3 client to Spring Cloud AWS. It's early stage but looks promising so far: https://github.com/awspring/spring-cloud-aws/blob/3.0.x-s3/spring-cloud-aws-s3-parent/spring-cloud-aws-s3/src/main/java/io/awspring/cloud/s3/CrossRegionS3Client.java

Basic idea is that every region-specific method is wrapped in a try catch that on http status 301 uses client created for the region where the bucket got created. Clients are cached both by region and also by bucket name (to speed up subsequent calls for the same bucket). Implementation is a bit Spring specific as we use LRU cache from spring-core, but if needed this can be replaced with something else or abstracted out.

Repetitive chunk of this class is generated with JavaParser. If you're interested this is a very sketchy implementation:

• https://github.com/awspring/spring-cloud-aws/blob/3.0.x-s3/spring-cloud-aws-s3-parent/spring-cloud-aws-s3-codegen/src/main/java/io/awspring/cloud/s3/codegen/CrossRegionS3ClientGenerator.java
• https://github.com/awspring/spring-cloud-aws/blob/3.0.x-s3/spring-cloud-aws-s3-parent/spring-cloud-aws-s3-codegen/src/main/java/io/awspring/cloud/s3/codegen/CrossRegionS3ClientTemplate.java

[zoewangg]

This feature has been released in 2.20.99. You can enable it by setting crossRegionAccessEnabled to true on the S3 client builder.
Example:

S3AsyncClient client = S3AsyncClient.builder()
                                    .crossRegionAccessEnabled(true)
                                    .build();

I'll go ahead and close this issue. As always, feedback is welcome! :)

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

[dasmowenator]

So the addition of .crossRegionAccessEnabled(true) is nice... but it still doesn't work with opt-in regions, even if your account is opted into the given region. For example, if I create an S3 client in the us-east-1 region and then try to head an object in an af-south-1 bucket that I own, it will result in a 400 error. Or if I create an S3 client in the af-south-1 region and then try to head an object in an us-east-1 bucket that I own, it will also result in a 400 error. The SDKv1 client didn't have this problem...