Remove crypto dependency from sdk #2952

sbiscigl · 2024-05-09T15:10:14Z

Description of changes:

Removes the direct dependency on libcrypto from the SDK and uses c-cal from crt instead simplifying the dependency set in the SDK.

Mostly borrowed from pull/2460 with bug fixes.

Check all that applies:

Did a review by yourself.
Added proper tests to cover this PR. (If tests are not applicable, explain.)
Checked if this PR is a breaking (APIs have been changed) change.
Checked if this PR will not introduce cross-platform inconsistent behavior.
Checked if this PR would require a ReadMe/Wiki update.

Check which platforms you have built SDK on to verify the correctness of this PR.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

CMakeLists.txt

src/aws-cpp-sdk-core/include/aws/core/utils/Array.h

cmake/external_dependencies.cmake

src/aws-cpp-sdk-core/include/aws/core/utils/Array.h

src/aws-cpp-sdk-core/include/aws/core/utils/base64/Base64.h

src/aws-cpp-sdk-core/include/aws/core/utils/crypto/Cipher.h

src/aws-cpp-sdk-core/include/aws/core/utils/crypto/SecureRandom.h

src/aws-cpp-sdk-core/include/aws/core/utils/crypto/crt/CRTHash.h

DmitriyMusatkin · 2024-05-20T16:58:18Z

src/aws-cpp-sdk-core/source/auth/signer/AWSAuthEventStreamV4Signer.cpp

-    auto hashResult = m_hash.Calculate(canonicalRequestString);
-    if (!hashResult.IsSuccess())
+    auto sha256Digest = HashingUtils::CalculateSHA256(canonicalRequestString);
+    if (sha256Digest.GetLength() == 0)


its still computing result, no? why does this have to change?

CalculateSHA256 returns a byte buf, which according to the CRT API is empty if it "failed". since this utility function hides away cipher state we use the empty/non-empty signal to determine if it was successfully calculated.

DmitriyMusatkin · 2024-05-20T18:17:06Z

src/aws-cpp-sdk-core/source/auth/signer/AWSAuthV4Signer.cpp

-    auto hashResult = m_HMAC->Calculate(ByteBuffer((unsigned char*)simpleDate.c_str(), simpleDate.length()),
-            ByteBuffer((unsigned char*)signingKey.c_str(), signingKey.length()));
+    auto kDate = HashingUtils::CalculateSHA256HMAC(
+        ByteBuffer((unsigned char *) simpleDate.c_str(), simpleDate.length()),


can we just add bytebuffer from string constructor?

ByteBuffer is an Array, but yeah that would be a good convenience function let me add that

DmitriyMusatkin · 2024-05-20T18:19:45Z

src/aws-cpp-sdk-core/source/utils/HashingUtils.cpp

    if (str.size() == 0)
    {
+        Sha256 hash;


do we even need object for this anymore? can we just make it a static helper?

so in theory yes, we could make a static helper function but it would require a major re-write of of how hashing works. specifically in Sha256 we separate implementation using pimpl using a unique ptr to a Hash type. because of this pimpl indirection we dont know the underlying type and invoke it using its virtual function. We could of course remove this pimpl indirection but the delta of that change would prove to be much more than and just declaring a temp hash var and using the vtable. IMO it would create a diff that would make this change more risky than it already is.

we could just have a static method in Sha256 that creates an instance and calls hash on it, instead of doing it all over the place? or have oneshot variant that resets state to working after hash? but whatever, not something to lose sleep over.

src/aws-cpp-sdk-core/source/utils/crypto/crt/CRTSecureRandomBytes.cpp

DmitriyMusatkin · 2024-05-20T18:22:52Z

src/aws-cpp-sdk-core/source/utils/crypto/crt/CRTSymmetricCipher.cpp

+                {
+                    return {std::move(resultBuffer)};
+                }
+                Crt::ByteBufDelete(resultBuffer);


how does error handling work here? we just return empty cryptoBuffer and nothing else?

yeah thats all from the original API which heavily relied on the virtual operator bool() const { return Good(); } operator to check if the cipher was in a erroronus state or not. which is why i change in this PR to catpure the state of being finished or ready. The error handling here would be to check the ciphers boolean operator before using the return value per the original API.

SergeyRyabinin · 2024-05-22T17:32:50Z

src/aws-cpp-sdk-core/include/aws/core/utils/Array.h

+
+            CryptoBuffer& operator=(Crt::ByteBuf&& other) noexcept
+            {
+                m_capacity = other.len;


add a check that other != parent of this
(i.e. static_cast<Crt::ByteBuf*>(this) != &other) or smth like this
otherwise you risk erasing the parent

CryptoBuffer != Crt::ByteBuf we actually want to delete the parent unconditionally as we are taking ownership of the underlying buffer from the bytebuf. since this is a move operation other is invalid after this operation and CryptoBuffer should always take owneship from parent, and it cannot move from self as they are different types.

SergeyRyabinin · 2024-05-22T17:40:04Z

src/aws-cpp-sdk-core/include/aws/core/utils/crypto/crt/CRTHash.h

+
+                HashResult GetHash() override;
+
+                // @private but made public for the sake of make_shared.


have you tried to fix perfect forwarding in MakeShared / make_shared?

discussed offline, removing comment, its just a move ctor from the underlying type

src/aws-cpp-sdk-core/source/utils/HashingUtils.cpp

src/aws-cpp-sdk-core/source/utils/crypto/CryptoBuf.cpp

src/aws-cpp-sdk-core/source/utils/crypto/crt/CRTHash.cpp

src/aws-cpp-sdk-core/source/utils/crypto/crt/CRTSecureRandomBytes.cpp

SergeyRyabinin · 2024-05-22T18:01:58Z

tests/aws-cpp-sdk-mediastore-data-integration-tests/MediaStoreDataTest.cpp

@@ -109,7 +109,7 @@ namespace
        static Aws::String GetTestContainerName()
        {
            // "-" is not allowed in container name, convert uuid to its hex.
-            static const std::string suffix = HashingUtils::HexEncode(ByteBuffer(Aws::Utils::UUID::RandomUUID())).c_str();
+            static const std::string suffix = HashingUtils::HexEncode(ByteBuffer(Aws::Utils::UUID::RandomUUID().operator ByteBuffer())).c_str();


why did you have to change this?
it seems to be an API backward incompatible change.

This is because of @DmitriyMusatkin 's requested change for adding a string to array ctor.. UUID is a special class in that it has a
operator Aws::String() const
AND
operator ByteBuffer() const

so when we convert a UUID to byte buf the ctor is ambigious now because there is a ctor with both string and byte buf. Is this a breaking API change? sort of? if a class has a operator defined for both String and ByteBuf and used in a constructor for Array. I think we can rationalize in making this change as we are the only people likely doing that.

--------- Co-authored-by: Jonathan M. Henson <henso@amazon.com>

sbiscigl force-pushed the remove-crypto-dep branch 7 times, most recently from e61128a to 8f4d559 Compare May 13, 2024 20:03

jmklix mentioned this pull request May 15, 2024

libcrypto linking issue with AWS SDK 1.11.285 #2962

Closed

jmklix linked an issue May 15, 2024 that may be closed by this pull request

libcrypto linking issue with AWS SDK 1.11.285 #2962

Closed

sbiscigl force-pushed the remove-crypto-dep branch 5 times, most recently from 4e0fa6b to 2fc815e Compare May 20, 2024 14:46

DmitriyMusatkin reviewed May 20, 2024

View reviewed changes

sbiscigl force-pushed the remove-crypto-dep branch from 2fc815e to fdcf989 Compare May 20, 2024 17:54

DmitriyMusatkin reviewed May 20, 2024

View reviewed changes

sbiscigl force-pushed the remove-crypto-dep branch 3 times, most recently from 3e07631 to 243722c Compare May 22, 2024 14:52

DmitriyMusatkin approved these changes May 22, 2024

View reviewed changes