Update CRT libraries and set operation_name for DEFAULT meta-requests (…

…#935)

* Update CRT submodules to latest releases

Submodule mountpoint-s3-crt-sys/crt/aws-c-cal 96c47e3..11fc684:
  > Make AES GCM more consistent cross platform (#189)
  > Pin AWS-LC until it's fixed for manylinux1 (#188)
  > Implement runtime check on libcrypto linkage (#186)
  > clang-format 18 (#187)
Submodule mountpoint-s3-crt-sys/crt/aws-c-common 06cf4d8..6d974f9:
  > cbor support  (#1131)
  > Fix default thread options for windows to not pin to any cpu_id (#1126)
  > Use CBMC 6.0.0 (#1128)
  > latest_submodules.py uses AWS-LC-FIPS releases in aws-crt-java (#1125)
  > Use CBMC version 5.95.1 (#1124)
  > clang-format 18 (#1113)
  > disable optimization was not working (#1123)
  > Fix memtracer bad assumptions on the size of stack trace (#1122)
Submodule mountpoint-s3-crt-sys/crt/aws-c-s3 6588f9a..cb431ba:
  > test_helper.py improvements (#442)
  > Fix shutdown_callback or returning NULL contract for meta_request (#440)
  > BREAKING CHANGE: operation_name must be set for DEFAULT meta-requests (#439)
  > clang-format 18 (#438)
  > Auto - Update S3 Ruleset & Partition (#436)
Submodule mountpoint-s3-crt-sys/crt/aws-lc 92bf532..4368aaa:
  > Fix for loading JCA stripped private keys (#1658)
  > Prepare for release v1.30.1 (#1657)
  > Revert  `_CET_ENDBR` (#1656)
  > Close FD in Snapsafe test function (#1649)
  > Prepare for release v1.30.0 (#1646)
  > Snapsafe-type uniqueness breaking event detection (#1640)
  > Add EVP_md_null and SSL_set_ciphersuites (#1637)
  > Add de-randomized ML-KEM modes to experimental EVP API (#1578)
  > Patch for OpenVPN certificate setting behavioral difference (#1643)
  > Require newer assembler for _CET_ENDBR (#1641)
  > OpenVPN error codes, SSL_get_peer_signature_* funcs, and first patch file (#1584)
  > NIST.SP.800-56Cr2 One-Step Key Derivation (#1607)
  > Upstream merge 2024-06-13 (#1636)
  > More minor symbols for Ruby support (#1581)
  > Add support for NETSCAPE_SPKI_print (#1624)
  > align gcc version with curl's CI (#1633)
  > Fix spelling nits
  > Generated ASM files
  > Add Intel Indirect Branch Tracking support.
  > [EC] Unify point addition for P-256/384/521 (#1602)
  > Upstream merge 2024 06 03 (#1621)
  > Fix AES key size for AES256 in ABI test (#1629)
  > Move SSL_CIPHER_get_version test to SSLVersionTest.Version (#1631)
  > Use 'nasm' not 'yasm' (#1630)
  > Prepare for release 1.29.0 (#1626)
  > Implement SSL_CIPHER_get_version for recent TLS versions (#1627)
  > Add integration tests for OpenSSL-linking 3p modules (#1587)
  > Prevent non-constant-time code in Kyber-R3 and ML-KEM implementation (#1619)
  > Update ec2-test-framework to use gv2 (#1623)
  > Script for creating compilation database (#1617)
  > Fixes for building with `-pedantic` (#1608)
  > Fix SSL_BUILD_CHAIN_FLAG_IGNORE_ERROR behavior (#1620)
  > Update for FIPS documentation (#1610)
  > Disable CI for gcc-14/FIPS until relocation issue is resolved (#1622)
  > Add support for ocsp get id (#1609)
  > Add libevent to GitHub integration CI (#1615)
  > Upstream merge 2024 05 17 (#1600)
  > add back ASN1_dup with tests (#1591)
  > Remove special aarch64 valgrind logic (#1618)
  > Fix NTP integ test (#1616)
  > Pin aws-lc-rs integ to nightly-2024-05-22 (#1612)
  > Cleanse the right amount of bytes in HMAC. (#1613)
  > add support for X509_CRL_http_nbio (#1596)
  > Add `all_fuzz_tests` build target (#1605)
  > Fix mariadb ssl_crl patch (#1606)
Submodule mountpoint-s3-crt-sys/crt/s2n-tls 6d92b46..073c7b4:
  > bug: Fixing bash error (#4624)
  > chore: make cbmc proof build more strict by adding -Werror flag (#4606)
  > Perform 2-RTT Handshake to upgrade to PQ when possible (#4526)
  > test(bindings/s2n-tls): refactor testing::s2n-tls tests (#4613)
  > docs: add timeout note to blinding delay docs (#4621)
  > docs: Add back suggested FIPS + TLS1.3 policy (#4605)
  > ci: shallow clone musl repo (#4611)
  > example(bindings): add async ConfigResolver (#4477)
  > chore: use CBMC version 5.95.1 (#4586)
  > s2n-tls rust binding: expose selected application protocol (#4599)
  > test: add pcap testing crate (#4604)
  > testing(bindings): add new test helper (#4596)
  > chore(bindings): fix shebang in generate.sh (#4603)
  > fix(s2n_session_ticket_test): correct clock mocking (#4602)
  > Fix: update default cert chain for unit tests (#4582)
  > refactor(binding): more accurate naming for const str helper (#4601)
  > fix: error rather than empty cipher suites (#4597)
  > chore: update s2n_stuffer_printf CBMC harness (#4531)
  > ci(nix): Fix integ pq test in a devShell (#4576)
  > feature: new compatibility-focused security policy preferring ECDSA (#4579)
  > compliance: update generate_report.sh to point to compliance directory (#4588)
  > ci: fix cppcheck errors (#4589)
  > chore: cleanup duplicate duvet citations (#4587)
  > Merge pull request from GHSA-52xf-5p2m-9wrv
  > chore(bindings): release 0.2.7 (#4580)
  > fix: Validate received signature algorithm in EVP verify (#4574)
  > refactor: add try_compile feature probe for RSA-PSS signing (#4569)
  > feat: Configurable blinding (#4562)
  > docs: document s2n_cert_auth_type behavior (#4454)
  > fix: init implicit iv for serialization feature (#4572)
  > [Nix] adjust pytest retrys (#4558)
  > fix: cert verify test fix (#4545)
  > fix: update default security policies (#4523)
  > feat(bindings): Associate an application context with a Connection (#4563)
  > chore(bindings): version bump (#4566)
  > Additional test cases for s2n_constant_time_equals() (#4559)
  > test: backwards compatibility test for the serialization feature (#4548)
  > chore(bench): upgrade rustls (#4554)

Signed-off-by: Alessandro Passaro <alexpax@amazon.com>

* Try to reduce package size

Signed-off-by: Alessandro Passaro <alexpax@amazon.com>

* Set operation_name when using MetaRequestType::Default

Signed-off-by: Alessandro Passaro <alexpax@amazon.com>

* Introduce S3Operation type

Signed-off-by: Alessandro Passaro <alexpax@amazon.com>

---------

Signed-off-by: Alessandro Passaro <alexpax@amazon.com>
Co-authored-by: Alessandro Passaro <alexpax@amazon.com>

mountpoint-s3-client/src/s3_crt_client.rs

@@ -444,7 +444,7 @@ impl S3CrtClientInner {
         })
     }
 
-    fn new_meta_request_options(message: S3Message, meta_request_type: MetaRequestType) -> MetaRequestOptions {
+    fn new_meta_request_options(message: S3Message, operation: S3Operation) -> MetaRequestOptions {
         let mut options = MetaRequestOptions::new();
         if let Some(checksum_config) = message.checksum_config {
             options.checksum_config(checksum_config);
@@ -455,7 +455,10 @@ impl S3CrtClientInner {
         options
             .message(message.inner)
             .endpoint(message.uri)
-            .request_type(meta_request_type);
+            .request_type(operation.meta_request_type());
+        if let Some(operation_name) = operation.operation_name() {
+            options.operation_name(operation_name);
+        }
         options
     }
 
@@ -470,15 +473,15 @@ impl S3CrtClientInner {
     fn make_meta_request<T: Send + 'static, E: std::error::Error + Send + 'static>(
         &self,
         message: S3Message,
-        meta_request_type: MetaRequestType,
+        operation: S3Operation,
         request_span: Span,
         on_headers: impl FnMut(&Headers, i32) + Send + 'static,
         on_body: impl FnMut(u64, &[u8]) + Send + 'static,
         on_finish: impl FnOnce(&MetaRequestResult) -> Result<T, Option<ObjectClientError<E, S3RequestError>>>
             + Send
             + 'static,
     ) -> Result<S3HttpRequest<T, E>, S3RequestError> {
-        let options = Self::new_meta_request_options(message, meta_request_type);
+        let options = Self::new_meta_request_options(message, operation);
         self.make_meta_request_from_options(options, request_span, |_| {}, on_headers, on_body, on_finish)
     }
 
@@ -669,11 +672,11 @@ impl S3CrtClientInner {
     fn make_simple_http_request<E: std::error::Error + Send + 'static>(
         &self,
         message: S3Message,
-        request_type: MetaRequestType,
+        operation: S3Operation,
         request_span: Span,
         on_error: impl FnOnce(&MetaRequestResult) -> Option<E> + Send + 'static,
     ) -> Result<S3HttpRequest<Vec<u8>, E>, S3RequestError> {
-        let options = Self::new_meta_request_options(message, request_type);
+        let options = Self::new_meta_request_options(message, operation);
         self.make_simple_http_request_from_options(options, request_span, |_| {}, on_error, |_, _| ())
     }
 
@@ -744,6 +747,42 @@ impl S3CrtClientInner {
     }
 }
 
+/// S3 operation supported by this client.
+#[derive(Debug, Clone, Copy)]
+enum S3Operation {
+    DeleteObject,
+    GetObject,
+    GetObjectAttributes,
+    HeadBucket,
+    HeadObject,
+    ListObjects,
+    PutObject,
+}
+
+impl S3Operation {
+    /// The [MetaRequestType] to use for this operation.
+    fn meta_request_type(&self) -> MetaRequestType {
+        match self {
+            S3Operation::GetObject => MetaRequestType::GetObject,
+            S3Operation::PutObject => MetaRequestType::PutObject,
+            _ => MetaRequestType::Default,
+        }
+    }
+
+    /// The operation name to set when configuring a request, if required.
+    fn operation_name(&self) -> Option<&'static str> {
+        match self {
+            S3Operation::DeleteObject => Some("DeleteObject"),
+            S3Operation::GetObject => None,
+            S3Operation::GetObjectAttributes => Some("GetObjectAttributes"),
+            S3Operation::HeadBucket => Some("HeadBucket"),
+            S3Operation::HeadObject => Some("HeadObject"),
+            S3Operation::ListObjects => Some("ListObjectsV2"),
+            S3Operation::PutObject => None,
+        }
+    }
+}
+
 /// A HTTP message to be sent to S3. This is a wrapper around a plain HTTP message, except that it
 /// helps us correctly configure the endpoint and "Host" header to handle both path-style and
 /// virtual-hosted-style addresses. The `path_prefix` is appended to the front of all paths, and

mountpoint-s3-client/src/s3_crt_client/delete_object.rs

@@ -1,10 +1,10 @@
 use std::ops::Deref;
 use std::os::unix::prelude::OsStrExt;
 
-use mountpoint_s3_crt::s3::client::{MetaRequestResult, MetaRequestType};
+use mountpoint_s3_crt::s3::client::MetaRequestResult;
 
 use crate::object_client::{DeleteObjectError, DeleteObjectResult, ObjectClientResult};
-use crate::s3_crt_client::{S3CrtClient, S3RequestError};
+use crate::s3_crt_client::{S3CrtClient, S3Operation, S3RequestError};
 
 impl S3CrtClient {
     /// Create and begin a new DeleteObject request.
@@ -26,7 +26,7 @@ impl S3CrtClient {
                 .map_err(S3RequestError::construction_failure)?;
 
             self.inner
-                .make_simple_http_request(message, MetaRequestType::Default, span, parse_delete_object_error)?
+                .make_simple_http_request(message, S3Operation::DeleteObject, span, parse_delete_object_error)?
         };
 
         let _body = request.await?;

mountpoint-s3-client/src/s3_crt_client/get_object.rs

@@ -9,13 +9,11 @@ use futures::channel::mpsc::UnboundedReceiver;
 use futures::Stream;
 use mountpoint_s3_crt::common::error::Error;
 use mountpoint_s3_crt::http::request_response::Header;
-use mountpoint_s3_crt::s3::client::{MetaRequestResult, MetaRequestType};
+use mountpoint_s3_crt::s3::client::MetaRequestResult;
 use pin_project::pin_project;
 
 use crate::object_client::{ETag, GetBodyPart, GetObjectError, ObjectClientError, ObjectClientResult};
-use crate::s3_crt_client::{S3CrtClient, S3HttpRequest, S3RequestError};
-
-use super::GetObjectRequest;
+use crate::s3_crt_client::{GetObjectRequest, S3CrtClient, S3HttpRequest, S3Operation, S3RequestError};
 
 impl S3CrtClient {
     /// Create and begin a new GetObject request. The returned [GetObjectRequest] is a [Stream] of
@@ -63,7 +61,7 @@ impl S3CrtClient {
 
         let request = self.inner.make_meta_request(
             message,
-            MetaRequestType::GetObject,
+            S3Operation::GetObject,
             span,
             |_, _| (),
             move |offset, data| {

mountpoint-s3-client/src/s3_crt_client/get_object_attributes.rs

@@ -2,17 +2,14 @@ use std::ops::Deref;
 use std::os::unix::prelude::OsStrExt;
 use std::str::FromStr;
 
-use mountpoint_s3_crt::{
-    http::request_response::Header,
-    s3::client::{MetaRequestResult, MetaRequestType},
-};
+use mountpoint_s3_crt::{http::request_response::Header, s3::client::MetaRequestResult};
 use thiserror::Error;
 
 use crate::object_client::{
     Checksum, GetObjectAttributesError, GetObjectAttributesParts, GetObjectAttributesResult, ObjectAttribute,
     ObjectClientError, ObjectClientResult, ObjectPart,
 };
-use crate::s3_crt_client::{S3CrtClient, S3RequestError};
+use crate::s3_crt_client::{S3CrtClient, S3Operation, S3RequestError};
 
 #[derive(Error, Debug)]
 #[non_exhaustive]
@@ -152,7 +149,7 @@ impl S3CrtClient {
 
             self.inner.make_simple_http_request(
                 message,
-                MetaRequestType::Default,
+                S3Operation::GetObjectAttributes,
                 span,
                 parse_get_object_attributes_error,
             )?

mountpoint-s3-client/src/s3_crt_client/head_bucket.rs

@@ -1,6 +1,5 @@
 use crate::object_client::ObjectClientResult;
-use crate::s3_crt_client::{S3CrtClient, S3RequestError};
-use mountpoint_s3_crt::s3::client::MetaRequestType;
+use crate::s3_crt_client::{S3CrtClient, S3Operation, S3RequestError};
 use thiserror::Error;
 
 /// Errors returned by a [`head_bucket`](S3CrtClient::head_bucket) request.
@@ -27,7 +26,7 @@ impl S3CrtClient {
                 let span = request_span!(self.inner, "head_bucket");
 
                 self.inner
-                    .make_simple_http_request(message, MetaRequestType::Default, span, |request_result| {
+                    .make_simple_http_request(message, S3Operation::HeadBucket, span, |request_result| {
                         match request_result.response_status {
                             404 => Some(HeadBucketError::NoSuchBucket),
                             _ => None,

mountpoint-s3-client/src/s3_crt_client/head_object.rs

@@ -4,7 +4,7 @@ use std::sync::{Arc, Mutex};
 
 use lazy_static::lazy_static;
 use mountpoint_s3_crt::http::request_response::{Headers, HeadersError};
-use mountpoint_s3_crt::s3::client::{MetaRequestResult, MetaRequestType};
+use mountpoint_s3_crt::s3::client::MetaRequestResult;
 use regex::Regex;
 use thiserror::Error;
 use time::format_description::well_known::Rfc2822;
@@ -14,7 +14,7 @@ use tracing::error;
 use crate::object_client::{
     HeadObjectError, HeadObjectResult, ObjectClientError, ObjectClientResult, ObjectInfo, RestoreStatus,
 };
-use crate::s3_crt_client::{S3CrtClient, S3RequestError};
+use crate::s3_crt_client::{S3CrtClient, S3Operation, S3RequestError};
 
 #[derive(Error, Debug)]
 #[non_exhaustive]
@@ -133,7 +133,7 @@ impl S3CrtClient {
 
             self.inner.make_meta_request(
                 message,
-                MetaRequestType::Default,
+                S3Operation::HeadObject,
                 span,
                 move |headers, _status| {
                     let mut header = header1.lock().unwrap();

mountpoint-s3-client/src/s3_crt_client/list_objects.rs

@@ -3,7 +3,7 @@ use std::os::unix::prelude::OsStrExt;
 use std::str::FromStr;
 
 use mountpoint_s3_crt::http::request_response::Header;
-use mountpoint_s3_crt::s3::client::{MetaRequestResult, MetaRequestType};
+use mountpoint_s3_crt::s3::client::MetaRequestResult;
 use thiserror::Error;
 use time::format_description::well_known::Rfc3339;
 use time::OffsetDateTime;
@@ -12,7 +12,7 @@ use tracing::error;
 use crate::object_client::{
     ListObjectsError, ListObjectsResult, ObjectClientError, ObjectClientResult, ObjectInfo, RestoreStatus,
 };
-use crate::s3_crt_client::{S3CrtClient, S3RequestError};
+use crate::s3_crt_client::{S3CrtClient, S3Operation, S3RequestError};
 
 #[derive(Error, Debug)]
 #[non_exhaustive]
@@ -188,7 +188,7 @@ impl S3CrtClient {
             );
 
             self.inner
-                .make_simple_http_request(message, MetaRequestType::Default, span, parse_list_objects_error)?
+                .make_simple_http_request(message, S3Operation::ListObjects, span, parse_list_objects_error)?
         };
 
         let body = body.await?;

mountpoint-s3-client/src/s3_crt_client/put_object.rs

@@ -2,15 +2,16 @@ use std::sync::{Arc, Mutex};
 use std::time::Instant;
 
 use crate::object_client::{ObjectClientResult, PutObjectError, PutObjectParams, PutObjectRequest, PutObjectResult};
-use crate::s3_crt_client::{emit_throughput_metric, PutObjectTrailingChecksums, S3CrtClient, S3RequestError};
+use crate::s3_crt_client::{
+    emit_throughput_metric, PutObjectTrailingChecksums, S3CrtClient, S3CrtClientInner, S3HttpRequest, S3Operation,
+    S3RequestError,
+};
 use async_trait::async_trait;
 use futures::channel::oneshot;
 use mountpoint_s3_crt::http::request_response::{Header, Headers};
-use mountpoint_s3_crt::s3::client::{ChecksumConfig, MetaRequestType, RequestType, UploadReview};
+use mountpoint_s3_crt::s3::client::{ChecksumConfig, RequestType, UploadReview};
 use tracing::error;
 
-use super::{S3CrtClientInner, S3HttpRequest};
-
 const SSE_TYPE_HEADER_NAME: &str = "x-amz-server-side-encryption";
 const SSE_KEY_ID_HEADER_NAME: &str = "x-amz-server-side-encryption-aws-kms-key-id";
 
@@ -65,7 +66,7 @@ impl S3CrtClient {
         let on_headers = move |headers: &Headers, _: i32| {
             *response_headers_writer.lock().unwrap() = Some(headers.clone());
         };
-        let mut options = S3CrtClientInner::new_meta_request_options(message, MetaRequestType::PutObject);
+        let mut options = S3CrtClientInner::new_meta_request_options(message, S3Operation::PutObject);
         options.send_using_async_writes(true);
         options.on_upload_review(move |review| callback.invoke(review));
 

mountpoint-s3-crt-sys/Cargo.toml

@@ -22,6 +22,7 @@ exclude = [
     "crt/aws-lc/crypto/fipsmodule/*/*_tests.txt",
     "crt/aws-lc/crypto/fipsmodule/policydocs/*",
     "crt/aws-lc/crypto/fipsmodule/sha/testvectors/*",
+    "crt/aws-lc/crypto/hpke/hpke_test_vectors.txt",
     "crt/aws-lc/crypto/hpke/test-vectors.json",
     "crt/aws-lc/crypto/kyber/kat/*",
     "crt/aws-lc/crypto/x509/*_test.cc",
@@ -33,6 +34,7 @@ exclude = [
     "crt/aws-lc/third_party/googletest/test/*",
     "crt/aws-lc/third_party/wycheproof_testvectors/*",
     "crt/aws-lc/util/fipstools/acvp/acvptool/test/*",
+    "crt/aws-lc/util/fipstools/delocate/testdata/*",
     "crt/s2n-tls/compliance/*",
     "crt/s2n-tls/scram/SCRAM_paper.pdf",
     "**/.github/**",

mountpoint-s3-crt/src/s3/client.rs

@@ -279,6 +279,15 @@ impl MetaRequestOptions {
         Self(options)
     }
 
+    /// Set the S3 operation name of the request.
+    pub fn operation_name(&mut self, operation_name: &'static str) -> &mut Self {
+        // SAFETY: we aren't moving out of the struct.
+        let options = unsafe { Pin::get_unchecked_mut(Pin::as_mut(&mut self.0)) };
+        // SAFETY: `operation_name` has a static lifetime.
+        options.inner.operation_name = unsafe { operation_name.as_aws_byte_cursor() };
+        self
+    }
+
     /// Set the message of the request.
     pub fn message(&mut self, message: Message) -> &mut Self {
         // SAFETY: we aren't moving out of the struct.