Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HADOOP-18073. Adds in s3 client config and updates list. #4706

Conversation

ahmarsuhail
Copy link
Contributor

@ahmarsuhail ahmarsuhail commented Aug 5, 2022

This provides the first set of changes for upgrading the s3 client. It configures the s3 client and updates the list operation.

Configuring the client

The createAwsConf method is now split into:
createClientConfigBuilder // sets request timeout, user agent
createHttpClientBuilder // sets max connections, connection timeout, socket timeout
createProxyConfigurationBuilder // sets proxy config, defined in table below

The table below lists the configurations S3A was using and what they now map to.
SDK V1 SDK V2
setMaxConnections httpClientBuilder.maxConnections
setProtocol The protocol is now HTTPS by default, and can only be modified by setting an HTTP endpoint on the client builder. This is done when setting the endpoint in getS3Endpoint()
setMaxErrorRetry createRetryPolicyBuilder
setConnectionTimeout httpClientBuilder.connectionTimeout
setSocketTimeout httpClientBuilder.socketTimeout
setRequestTimeout overrideConfigBuilder.apiCallAttemptTimeout
setSocketBufferSizeHints Not supported
setSignerOverride Not done yet
setProxyHost proxyConfigBuilder.endpoint
setProxyPort set when setting proxy host with .endpoint
setProxyUsername proxyConfigBuilder.username
setProxyPassword proxyConfigBuilder.password
setProxyDomain proxyConfigBuilder.ntlmDomain
setProxyWorkstation proxyConfigBuilder.ntlmWorkstation
setUserAgentPrefix overrideConfigBuilder.putAdvancedOption(SdkAdvancedClientOption.USER_AGENT_PREFIX, userAgent);
addHeader overrideConfigBuilder.putHeader
setUseThrottleRetries not supported

Endpoint and region configuration

Previously, if no endpoint and region was configured, fall back to using us-east-1 . Set withForceGlobalBucketAccessEnabled(true) which will allow access to buckets not in this region too.

Since the SDK V2 no longer supports cross region access, we need to set the region and endpoint of the bucket. The behaviour has now been changed to:

  • If no endpoint is specified, use s3.amazonaws.com.
  • When setting the endpoint, also set the protocol (HTTP or HTTPS)
  • When setting the region, first initiate a default S3 Client with region eu-west-2. Call headBucket using this client. If the bucket is also in eu-west-2, then this will return a successful response. Otherwise it will throw an error with status code 301 permanently moved. This error contains the region of the bucket in it’s header, which we can then use to configure the client.

Things not done yet:

  • Client side encryption - S3 encryption not supported in the SDK yet.
  • Signers - will be done separately
  • Setting request handlers - Will be done as part of auditor work
  • Metrics collection - Will need further investigation. SDK V2 has a metrics publisher, but could not find how to collect metrics.

Updating ListOperations

This was pretty straightforward. A few things to note:

  • I could not find a way to paginate listV1 using the SDK. So am doing this instead.
  • Etags in the list operation (and probably all other ops) are now returned with quotations, eg: “123”. SDKV1 returns etags without quotations so this causes equality failures in tests. Added in a temp fix for this here.
  • Requests are not wired up to the auditor yet which causes some tests to fail. We will do this when we’re closer to completing all operation updates. However I will do a dive deep on the auditor, how it works and what we’ll need to do next week.
  • Adding in MockMaker as this is need to mock final classes ListObjectsResponse & ListObjectsV2Result.

Failing Tests

The following tests are failing:

Test Failing Reason
ITestS3AContractEtag.testLocatedStatusAlsoHasEtag Fails because the head call returns Etag without quotations eg etag: 123, but new list returns with quotations eg etag: "123", so equality of two etags fails. (Not fixed by above etag fix as it's a contract test and so equality test happens in hadoop-common)
ITestS3ABucketExistence.testNoBucketProbing Errors are not translated yet, so SDKException does not get converted to UnknownStoreException
ITestS3AAWSCredentialsProvider.testBadCredentialsConstructor Fails because SDKV2 throws SdkException, will work once errors are handled and translated properly.
ITestAuditAccessChecks.testDirAccessDenied Fails because list requests are not wired up to auditor yet
ITestAuditAccessChecks.testMissingPathAccessFNFE Fails because list requests are not wired up to auditor yet
ITestAuditManager.testRequestHandlerBinding Fails because list requests are not wired up to auditor yet
ITestMarkerTool.testRunWrongBucket Throws NoSuchBucketException, which is not yet translated
ITestS3AConfiguration.testAutomaticProxyPortSelection Fails because SDKV2 throws SdkException, will work once errors are handled and translated properly.
ITestS3AConfiguration.testProxyConnection Fails because SDKV2 throws SdkException, will work once errors are handled and translated properly.

Copy link
Contributor

@dannycjones dannycjones left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking good!

I've added some suggestions - to put it simply, let's do whatever we can to reduce the diff and improve legibility to make it easier to review since we're going to end up touching a lot of files across S3A in the process of this upgrade.

Copy link
Contributor

@dannycjones dannycjones left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just some small feedback on the region determination code I missed on first review. After that, should be ready for +1.

Comment on lines 535 to 536
// build a s3 client with region eu-west-2 that can be used to get the region of the bucket.
S3Client s3Client = S3Client.builder().region(Region.EU_WEST_2)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

eu-west-2 / London seems a bit of a random decision.

Let's be intentional by using the AWS_GLOBAL region value - this will result in us-east-1. (Only works for commercial regions. Not sure if we can do anything better for China / US Gov Cloud?)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

documented in later commit. "us-east-1"/AWS_GLOBAL can't be used. thanks Ahmar

@ahmarsuhail
Copy link
Contributor Author

Thanks @dannycjones, have moved those things to constants.

RE using eu-west-1 and the region logic in general:

Not using us-east-1 as headBucket() fails with that region. This is because us-east-1 uses the endpoint s3.amazonaws.com, which resolves bucket.s3.amazonaws.com to the actual region the bucket is in. As the request is signed with us-east-1 and not the bucket's region, it fails. For more info, see this issue.

I'm not sure how the region logic should behave and how it should handle failures. Returning us-east-1 at the end here is not of much use, as without cross region access, if the region is configured incorrectly, any request to S3 will fail. Instead, to handle network failures etc, we should probably add some retry logic in this method.

I'm also not sure if this new region/endpoint logic is sufficient to handle third party stores, keen to know what other people think.

@ahmarsuhail
Copy link
Contributor Author

@steveloughran / @mukund-thakur , I've opened a PR with the first set of changes for the SDK upgrade, could you please take a look?

Copy link
Contributor

@dannycjones dannycjones left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 (non-binding)

Thanks Ahmar!

@hadoop-yetus
Copy link

🎊 +1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 53s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 1s No case conflicting files found.
+0 🆗 codespell 0m 1s codespell was not available.
+0 🆗 detsecrets 0m 1s detect-secrets was not available.
+0 🆗 xmllint 0m 1s xmllint was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 5 new or modified test files.
_ feature-HADOOP-18073-s3a-sdk-upgrade Compile Tests _
+0 🆗 mvndep 14m 32s Maven dependency ordering for branch
+1 💚 mvninstall 28m 26s feature-HADOOP-18073-s3a-sdk-upgrade passed
+1 💚 compile 25m 26s feature-HADOOP-18073-s3a-sdk-upgrade passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚 compile 22m 0s feature-HADOOP-18073-s3a-sdk-upgrade passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚 checkstyle 4m 25s feature-HADOOP-18073-s3a-sdk-upgrade passed
+1 💚 mvnsite 1m 51s feature-HADOOP-18073-s3a-sdk-upgrade passed
+1 💚 javadoc 1m 35s feature-HADOOP-18073-s3a-sdk-upgrade passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚 javadoc 1m 37s feature-HADOOP-18073-s3a-sdk-upgrade passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+0 🆗 spotbugs 0m 59s branch/hadoop-project no spotbugs output file (spotbugsXml.xml)
+1 💚 shadedclient 24m 9s branch has no errors when building and testing our client artifacts.
-0 ⚠️ patch 24m 36s Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
_ Patch Compile Tests _
+0 🆗 mvndep 0m 41s Maven dependency ordering for patch
+1 💚 mvninstall 0m 50s the patch passed
+1 💚 compile 24m 33s the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚 javac 24m 33s the patch passed
+1 💚 compile 22m 18s the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚 javac 22m 18s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 checkstyle 4m 17s the patch passed
+1 💚 mvnsite 1m 51s the patch passed
+1 💚 javadoc 1m 29s the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚 javadoc 1m 38s the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+0 🆗 spotbugs 0m 45s hadoop-project has no data from spotbugs
+1 💚 shadedclient 24m 14s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 0m 44s hadoop-project in the patch passed.
+1 💚 unit 3m 1s hadoop-aws in the patch passed.
+1 💚 asflicense 1m 9s The patch does not generate ASF License warnings.
220m 4s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4706/11/artifact/out/Dockerfile
GITHUB PR #4706
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint spotbugs checkstyle
uname Linux bc4dbb7be277 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision feature-HADOOP-18073-s3a-sdk-upgrade / 2ad9c81
Default Java Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4706/11/testReport/
Max. process+thread count 590 (vs. ulimit of 5500)
modules C: hadoop-project hadoop-tools/hadoop-aws U: .
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4706/11/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@apache apache deleted a comment from hadoop-yetus Aug 26, 2022
@apache apache deleted a comment from hadoop-yetus Aug 26, 2022
@apache apache deleted a comment from hadoop-yetus Aug 26, 2022
@apache apache deleted a comment from hadoop-yetus Aug 26, 2022
@apache apache deleted a comment from hadoop-yetus Aug 26, 2022
@apache apache deleted a comment from hadoop-yetus Aug 26, 2022
@apache apache deleted a comment from hadoop-yetus Aug 26, 2022
@apache apache deleted a comment from hadoop-yetus Aug 26, 2022
@apache apache deleted a comment from hadoop-yetus Aug 26, 2022
@apache apache deleted a comment from hadoop-yetus Aug 26, 2022
@apache apache deleted a comment from hadoop-yetus Aug 26, 2022
@ahmarsuhail
Copy link
Contributor Author

closing this, these changes will be part of a new PR I'll open once the feature branch is in sync with trunk.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants