Introduced support for the AWS_ENDPOINT_URL environment variable (#895)

* Adding support for AWS_ENDPOINT_URL.

Signed-off-by: Andres Santana <hernaa@amazon.com>

* Update CHANGELOG.md and CONFIGURATION.md

Signed-off-by: Andres Santana <hernaa@amazon.com>

* Using rust_fork to run the tests.

Signed-off-by: Andres Santana <hernaa@amazon.com>

* Addressing comments.

Signed-off-by: Andres Santana <hernaa@amazon.com>

* Adding changes from Danny.

2303bd8

Signed-off-by: Andres Santana <hernaa@amazon.com>

---------

Signed-off-by: Andres Santana <hernaa@amazon.com>

doc/CONFIGURATION.md

@@ -156,6 +156,11 @@ In most scenarios, Mountpoint automatically infers the appropriate Amazon S3 end
 
 If necessary, you can use the `--endpoint-url` command-line argument to fully override Mountpoint's endpoint detection. For example, the argument `--endpoint-url https://example.com` will force Mountpoint to send S3 requests to `example.com`. You may need to also use the `--region` flag to correctly specify the region to use for signing requests. By default, Mountpoint will use [virtual-hosted-style addressing](https://docs.aws.amazon.com/AmazonS3/latest/userguide/VirtualHosting.html) for the configured endpoint, and so will send requests to `https://docexamplebucket.example.com` if configured with `--endpoint-url https://example.com` and the bucket name `docexamplebucket`. To disable virtual-hosted-style addressing, use the `--force-path-style` command-line flag to instead send requests to `https://example.com/docexamplebucket/`.
 
+We also support the `AWS_ENDPOINT_URL` environment variable. The endpoint determination follows this order:
+- Use the CLI parameter `endpoint-url` if provided.
+- Use `AWS_ENDPOINT_URL` if provided.
+- Fallback to automically inferring the endpoint.
+
 ### Data encryption
 
 Amazon S3 supports a number of [server-side encryption types](https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingEncryption.html). Mountpoint supports reading and writing to buckets that are configured with Amazon S3 managed keys (SSE-S3), with AWS KMS keys (SSE-KMS), or with dual-layer encryption with AWS KMS keys (DSSE-KMS) as the default encryption method. It does not currently support reading objects encrypted with customer-provided keys (SSE-C).

mountpoint-s3-client/CHANGELOG.md

@@ -1,5 +1,7 @@
 ## Unreleased changes
 
+* Adding support for `AWS_ENDPOINT_URL` environment variable.
+
 ### Breaking changes
 
 * The `trailing_checksums` field of `PutObjectParams` is now an enum, with a new `ReviewOnly` option that allows disabling sending additional checksum headers to S3 while still computing them for use by `UploadReview` callbacks. ([#849](https://github.com/awslabs/mountpoint-s3/pull/849))

mountpoint-s3-client/src/s3_crt_client.rs

@@ -37,8 +37,8 @@ use tracing::{debug, error, trace, Span};
 
 use self::get_object::S3GetObjectRequest;
 use self::put_object::S3PutObjectRequest;
-use crate::endpoint_config::EndpointConfig;
 use crate::endpoint_config::EndpointError;
+use crate::endpoint_config::{self, EndpointConfig};
 use crate::object_client::*;
 use crate::user_agent::UserAgent;
 
@@ -303,6 +303,22 @@ impl S3CrtClientInner {
             None,
             None,
         );
+
+        let endpoint_config = match endpoint_config.get_endpoint() {
+            None => {
+                // No explicit endpoint was configured, let's try the environment variable
+                if let Ok(aws_endpoint_url) = std::env::var("AWS_ENDPOINT_URL") {
+                    debug!("using AWS_ENDPOINT_URL {}", aws_endpoint_url);
+                    let env_uri = Uri::new_from_str(&allocator, &aws_endpoint_url)
+                        .map_err(|e| EndpointError::InvalidUri(endpoint_config::InvalidUriError::CouldNotParse(e)))?;
+                    endpoint_config.endpoint(env_uri)
+                } else {
+                    endpoint_config
+                }
+            }
+            Some(_) => endpoint_config,
+        };
+
         client_config.express_support(true);
         client_config.signing_config(signing_config);
 
@@ -1087,6 +1103,7 @@ impl ObjectClient for S3CrtClient {
 #[cfg(test)]
 mod tests {
     use mountpoint_s3_crt::common::error::Error;
+    use rusty_fork::rusty_fork_test;
     use std::assert_eq;
 
     use super::*;
@@ -1155,6 +1172,63 @@ mod tests {
             .starts_with(expected_user_agent));
     }
 
+    fn assert_expected_host(expected_host: &str, endpoint_config: EndpointConfig) {
+        let config = S3ClientConfig {
+            endpoint_config,
+            ..Default::default()
+        };
+
+        let client = S3CrtClient::new(config).expect("create test client");
+
+        let mut message = client
+            .inner
+            .new_request_template("GET", "")
+            .expect("new request template expected");
+
+        let headers = message.inner.get_headers().expect("expected a block of HTTP headers");
+
+        let host_header = headers.get("Host").expect("Host header expected");
+        let host_header_value = host_header.value();
+
+        assert_eq!(host_header_value.to_string_lossy(), expected_host);
+    }
+
+    // run with rusty_fork to avoid issues with other tests and their env variables.
+    rusty_fork_test! {
+        #[test]
+        fn test_endpoint_favors_parameter_over_env_variable() {
+            let endpoint_uri = Uri::new_from_str(&Allocator::default(), "https://s3.us-west-2.amazonaws.com").unwrap();
+            let endpoint_config = EndpointConfig::new("region-place-holder").endpoint(endpoint_uri);
+            std::env::set_var("AWS_ENDPOINT_URL", "https://s3.us-east-1.amazonaws.com");
+            // even though we set the environment variable, the parameter takes precedence
+            assert_expected_host("s3.us-west-2.amazonaws.com", endpoint_config);
+        }
+
+        #[test]
+        fn test_endpoint_favors_env_variable() {
+            let endpoint_config = EndpointConfig::new("us-east-1");
+            std::env::set_var("AWS_ENDPOINT_URL", "https://s3.eu-west-1.amazonaws.com");
+            assert_expected_host("s3.eu-west-1.amazonaws.com", endpoint_config);
+        }
+
+        #[test]
+        fn test_endpoint_with_invalid_env_variable() {
+            let endpoint_config = EndpointConfig::new("us-east-1");
+            std::env::set_var("AWS_ENDPOINT_URL", "htp:/bad:url");
+            let config = S3ClientConfig {
+                endpoint_config,
+                ..Default::default()
+            };
+
+            let client = S3CrtClient::new(config);
+            match client {
+                Ok(_) => panic!("expected an error"),
+                Err(e) => assert_eq!(e.to_string().to_lowercase(), "invalid s3 endpoint"),
+            }
+        }
+
+    }
+
     /// Simple test to ensure the user agent header is correct even when prefix is not added
     #[test]
     fn test_user_agent_without_prefix() {

mountpoint-s3/CHANGELOG.md

@@ -9,6 +9,7 @@
 
 ### Other changes
 * The checksum algorithm to use for uploads to S3 can now be chosen with the `--upload-checksums <ALGORITHM>` command-line argument. The only supported values in this release are `crc32c` (the default, and the existing behavior) and `off`, which disables including checksums in uploads. The `off` value allows uploads to S3 implementations that do not support [additional checksums](https://aws.amazon.com/blogs/aws/new-additional-checksum-algorithms-for-amazon-s3/). This option defaults to `off` when the bucket name is an S3 on Outposts bucket access point (either an ARN or a bucket alias). ([#849](https://github.com/awslabs/mountpoint-s3/pull/849))
+* Adding support for `AWS_ENDPOINT_URL` environment variable.
 
 ## v1.6.0 (April 11, 2024)