CMake Config mode support for Curl #2717

pr0g · 2023-10-17T21:33:37Z

Describe the bug

Right now it does not seem possible to use a custom build of Curl built with CMake when it is installed in Config mode.

When building aws-sdk-cpp, CURL will be found in Config mode, but the include directories and library variables are not populated.

...
-- Http client: Curl
-- Found CURL: /path/to/build/lib/cmake/CURL/CURLConfig.cmake (found version "8.2.1-DEV")  
--   Curl include directory: 
--   Curl library: 
...

This is because in Config mode the variables ${<library>_INCLUDE_DIRS} and ${<library>_LIBRARIES} are not populated (in this case ${CURL_INCLUDE_DIRS} and ${CURL_LIBRARIES}).

There is a workaround for this (please see this experimental draft PR - #2718) where we use the CMake find_package command to find Curl, and then link against the Curl target in aws-sdk-cpp/src/aws-cpp-sdk-core/CMakeLists.txt

# cmake/external_dependencies.cmake
...
  find_package(CURL)
  if(NOT CURL_FOUND)
      message(FATAL_ERROR "Could not find curl")
  else()
...

# aws-cpp-sdk-core/CMakeLists.txt
...
target_link_libraries(
    ${PROJECT_NAME} PRIVATE 
    ${PLATFORM_DEP_LIBS} ${CLIENT_LIBS} ${CRYPTO_LIBS}
    ${AWS_SDK_ADDITIONAL_LIBRARIES}
    CURL::libcurl) # <-- added
...

I realize this is a super hacky change and is not fit for purpose as is, but it proves that this can work with a fairly minor modification.

Expected Behavior

aws-sdk-cpp can be built when using a build of CURL in Config mode (see CMake Finding packages for details).

Current Behavior

Currently aws-sdk-cpp does not build when using a build of CURL in Config mode.

Reproduction Steps

To reproduce this issue (tested on macOS) you can use this simple CMake project:

# CMakeLists.txt

cmake_minimum_required(VERSION 3.22)

project(aws-cpp-sdk-external)

include(ExternalProject)
include(ProcessorCount)

ProcessorCount(CPU_COUNT)

# Custom build of OpenSSL
ExternalProject_Add(
  openssl
  GIT_REPOSITORY https://github.com/openssl/openssl.git
  GIT_TAG openssl-3.2.0-alpha2
  GIT_SHALLOW ON
  UPDATE_COMMAND ""
  PREFIX ${CMAKE_CURRENT_BINARY_DIR}
  BINARY_DIR ${CMAKE_CURRENT_BINARY_DIR}/openssl/build/Debug
  CONFIGURE_COMMAND
    perl <SOURCE_DIR>/Configure no-docs no-tests no-legacy
    --prefix=${CMAKE_CURRENT_BINARY_DIR}
    --openssldir=ssl
  BUILD_COMMAND make -j ${CPU_COUNT}
  INSTALL_DIR ${CMAKE_CURRENT_BINARY_DIR}
  INSTALL_COMMAND make install_sw)

# Custom build of Curl
ExternalProject_Add(
  curl
  GIT_REPOSITORY https://github.com/curl/curl.git
  GIT_TAG curl-8_2_1
  DEPENDS openssl
  PREFIX ${CMAKE_CURRENT_BINARY_DIR}
  BINARY_DIR ${CMAKE_CURRENT_BINARY_DIR}/curl/build/Debug
  INSTALL_DIR ${CMAKE_CURRENT_BINARY_DIR}
  CMAKE_ARGS -DCMAKE_BUILD_TYPE=Debug -DCMAKE_INSTALL_PREFIX=<INSTALL_DIR>
             -DOPENSSL_ROOT_DIR=<INSTALL_DIR>
  CMAKE_CACHE_ARGS -DCMAKE_DEBUG_POSTFIX:STRING=d)

# Custom build of aws-cpp-sdk
ExternalProject_Add(
  aws-cpp-sdk
  GIT_REPOSITORY https://github.com/aws/aws-sdk-cpp.git
  GIT_TAG 1.11.181
  DEPENDS curl
  PREFIX ${CMAKE_CURRENT_BINARY_DIR}
  BINARY_DIR ${CMAKE_CURRENT_BINARY_DIR}/aws-cpp-sdk/build/Debug
  INSTALL_DIR ${CMAKE_CURRENT_BINARY_DIR}
  CMAKE_ARGS -DCMAKE_BUILD_TYPE=Debug
             -DCMAKE_INSTALL_PREFIX=<INSTALL_DIR>
             -DBUILD_ONLY=cognito-identity
             -DENABLE_TESTING=OFF
             -DAUTORUN_UNIT_TESTS=OFF
             -DFORCE_CURL=ON)

And then build with...

cmake -B build
cmake --build build

Possible Solution

Please see this experimental draft PR - #2718

This is by no means a full and proper solution but shows the general idea to start a discussion.

Additional Information/Context

I'd be very grateful to hear how this could be resolved. I'm happy to have a stab at making the above change more robust (handling scenarios where CURL shouldn't be linked etc...). Or maybe there is another approach where it is possible to populate the variables mentioned earlier with a CMake library in Config mode.

AWS CPP SDK version used

main

Compiler and Version used

Apple clang version 15.0.0 (clang-1500.0.40.1) (CMake 3.26.1)

Operating System and version

macOS Ventura 13.5.2

The text was updated successfully, but these errors were encountered:

SergeyRyabinin · 2023-10-17T22:07:37Z

Hi @pr0g ,

thank you for submitting this issue.
Last time I had a need to build with a custom built curl, I used the following argument:

-DCMAKE_PREFIX_PATH=$HOME/curl-install

I'm not a 100% expert in cmake, so maybe wrong here, is there a reason you are not able to use CMAKE_PREFIX_PATH when building the SDK?
We will review your PR too.

Best regards,
Sergey

pr0g · 2023-10-18T10:06:44Z

Hi @SergeyRyabinin,

Thank you for getting back to me so quickly about this.

Ah yes, the CMAKE_PREFIX_PATH usage is essentially what the ExternalProject_Add command is doing under the hood as far as I know. It is also possible to reproduce this issue by using CMAKE_PREFIX_PATH as you describe. To to do this, make a custom build of Curl (using CMake) and install it somewhere, and then run a command something like this:

# from aws-sdk-cpp root directory
cmake -B build -G Ninja -DCMAKE_PREFIX_PATH=path/to/curl/install \
  -DCMAKE_BUILD_TYPE=Debug -DBUILD_ONLY=cognito-identity \
  -DENABLE_TESTING=OFF -DAUTORUN_UNIT_TESTS=OFF \
  -DFORCE_CURL=ON
cmake --build build

The build fails at link time because the Curl libraries cannot be found (this is for the reason mentioned in the issue where ${CURL_INCLUDE_DIRS} and ${CURL_LIBRARIES} are not populated).

For completeness this is the CMakeLists.txt I used to build and install Curl

cmake_minimum_required(VERSION 3.22)

project(curl-external)

include(ExternalProject)
include(ProcessorCount)

ProcessorCount(CPU_COUNT)

ExternalProject_Add(
  openssl
  GIT_REPOSITORY https://github.com/openssl/openssl.git
  GIT_TAG openssl-3.2.0-alpha2
  GIT_SHALLOW ON
  UPDATE_COMMAND ""
  PREFIX ${CMAKE_CURRENT_BINARY_DIR}
  BINARY_DIR ${CMAKE_CURRENT_BINARY_DIR}/openssl/build/Debug
  CONFIGURE_COMMAND
    perl <SOURCE_DIR>/Configure no-docs no-tests no-legacy
    --prefix=${CMAKE_CURRENT_BINARY_DIR}
    --openssldir=ssl
  BUILD_COMMAND make -j ${CPU_COUNT}
  INSTALL_DIR ${CMAKE_CURRENT_BINARY_DIR}
  INSTALL_COMMAND make install_sw)

ExternalProject_Add(
  curl
  GIT_REPOSITORY https://github.com/curl/curl.git
  GIT_TAG curl-8_2_1
  DEPENDS openssl
  PREFIX ${CMAKE_CURRENT_BINARY_DIR}
  BINARY_DIR ${CMAKE_CURRENT_BINARY_DIR}/curl/build/Debug
  INSTALL_DIR ${CMAKE_CURRENT_BINARY_DIR}
  CMAKE_ARGS -DCMAKE_BUILD_TYPE=Debug -DCMAKE_INSTALL_PREFIX=<INSTALL_DIR>
             -DOPENSSL_ROOT_DIR=<INSTALL_DIR>
  CMAKE_CACHE_ARGS -DCMAKE_DEBUG_POSTFIX:STRING=d)

I just created a new directory for this and ran

cmake -B build -G Ninja
cmake --build build

And that build folder is what I pointed CMAKE_PREFIX_PATH at in the earlier command.

pr0g · 2023-10-18T10:11:09Z

Also for reference the issue manifests as this:

ld: Undefined symbols:
  _curl_easy_cleanup, referenced from:
      Aws::Http::CurlHandleContainer::~CurlHandleContainer() in CurlHandleContainer.cpp.o
      Aws::Http::CurlHandleContainer::DestroyCurlHandle(void*) in CurlHandleContainer.cpp.o
  _curl_easy_getinfo, referenced from:
      Aws::Http::CurlHttpClient::MakeRequest(std::__1::shared_ptr<Aws::Http::HttpRequest> const&, Aws::Utils::RateLimits::RateLimiterInterface*, Aws::Utils::RateLimits::RateLimiterInterface*) const in CurlHttpClient.cpp.o
      Aws::Http::CurlHttpClient::MakeRequest(std::__1::shared_ptr<Aws::Http::HttpRequest> const&, Aws::Utils::RateLimits::RateLimiterInterface*, Aws::Utils::RateLimits::RateLimiterInterface*) const in CurlHttpClient.cpp.o
      Aws::Http::CurlHttpClient::MakeRequest(std::__1::shared_ptr<Aws::Http::HttpRequest> const&, Aws::Utils::RateLimits::RateLimiterInterface*, Aws::Utils::RateLimits::RateLimiterInterface*) const in CurlHttpClient.cpp.o
      Aws::Http::CurlHttpClient::MakeRequest(std::__1::shared_ptr<Aws::Http::HttpRequest> const&, Aws::Utils::RateLimits::RateLimiterInterface*, Aws::Utils::RateLimits::RateLimiterInterface*) const in CurlHttpClient.cpp.o
      Aws::Http::CurlHttpClient::MakeRequest(std::__1::shared_ptr<Aws::Http::HttpRequest> const&, Aws::Utils::RateLimits::RateLimiterInterface*, Aws::Utils::RateLimits::RateLimiterInterface*) const in CurlHttpClient.cpp.o
      Aws::Http::CurlHttpClient::MakeRequest(std::__1::shared_ptr<Aws::Http::HttpRequest> const&, Aws::Utils::RateLimits::RateLimiterInterface*, Aws::Utils::RateLimits::RateLimiterInterface*) const in CurlHttpClient.cpp.o
      Aws::Http::CurlHttpClient::MakeRequest(std::__1::shared_ptr<Aws::Http::HttpRequest> const&, Aws::Utils::RateLimits::RateLimiterInterface*, Aws::Utils::RateLimits::RateLimiterInterface*) const in CurlHttpClient.cpp.o
      ...
  _curl_easy_init, referenced from:
      Aws::Http::CurlHandleContainer::CreateCurlHandleInPool() in CurlHandleContainer.cpp.o
  _curl_easy_pause, referenced from:
      CurlProgressCallback(void*, long, long, long, long) in CurlHttpClient.cpp.o
      CurlProgressCallback(void*, long, long, long, long) in CurlHttpClient.cpp.o
  _curl_easy_perform, referenced from:
      Aws::Http::CurlHttpClient::MakeRequest(std::__1::shared_ptr<Aws::Http::HttpRequest> const&, Aws::Utils::RateLimits::RateLimiterInterface*, Aws::Utils::RateLimits::RateLimiterInterface*) const in CurlHttpClient.cpp.o
  _curl_easy_reset, referenced from:
      Aws::Http::CurlHandleContainer::ReleaseCurlHandle(void*) in CurlHandleContainer.cpp.o
  _curl_easy_setopt, referenced from:
      Aws::Http::CurlHandleContainer::ReleaseCurlHandle(void*) in CurlHandleContainer.cpp.o
      Aws::Http::CurlHandleContainer::SetDefaultOptionsOnHandle(void*) in CurlHandleContainer.cpp.o
      Aws::Http::CurlHandleContainer::SetDefaultOptionsOnHandle(void*) in CurlHandleContainer.cpp.o
      Aws::Http::CurlHandleContainer::SetDefaultOptionsOnHandle(void*) in CurlHandleContainer.cpp.o
      Aws::Http::CurlHandleContainer::SetDefaultOptionsOnHandle(void*) in CurlHandleContainer.cpp.o
      Aws::Http::CurlHandleContainer::SetDefaultOptionsOnHandle(void*) in CurlHandleContainer.cpp.o
      Aws::Http::CurlHandleContainer::SetDefaultOptionsOnHandle(void*) in CurlHandleContainer.cpp.o
      Aws::Http::CurlHandleContainer::SetDefaultOptionsOnHandle(void*) in CurlHandleContainer.cpp.o
      Aws::Http::CurlHandleContainer::SetDefaultOptionsOnHandle(void*) in CurlHandleContainer.cpp.o
      Aws::Http::CurlHandleContainer::SetDefaultOptionsOnHandle(void*) in CurlHandleContainer.cpp.o
      Aws::Http::CurlHandleContainer::SetDefaultOptionsOnHandle(void*) in CurlHandleContainer.cpp.o
      ...
  _curl_easy_strerror, referenced from:
      Aws::Http::CurlHttpClient::MakeRequest(std::__1::shared_ptr<Aws::Http::HttpRequest> const&, Aws::Utils::RateLimits::RateLimiterInterface*, Aws::Utils::RateLimits::RateLimiterInterface*) const in CurlHttpClient.cpp.o
      Aws::Http::CurlHttpClient::MakeRequest(std::__1::shared_ptr<Aws::Http::HttpRequest> const&, Aws::Utils::RateLimits::RateLimiterInterface*, Aws::Utils::RateLimits::RateLimiterInterface*) const in CurlHttpClient.cpp.o
  _curl_global_cleanup, referenced from:
      Aws::Http::CurlHttpClient::CleanupGlobalState() in CurlHttpClient.cpp.o
  _curl_global_init, referenced from:
      Aws::Http::CurlHttpClient::InitGlobalState() in CurlHttpClient.cpp.o
  _curl_slist_append, referenced from:
      Aws::Http::CurlHttpClient::MakeRequest(std::__1::shared_ptr<Aws::Http::HttpRequest> const&, Aws::Utils::RateLimits::RateLimiterInterface*, Aws::Utils::RateLimits::RateLimiterInterface*) const in CurlHttpClient.cpp.o
      Aws::Http::CurlHttpClient::MakeRequest(std::__1::shared_ptr<Aws::Http::HttpRequest> const&, Aws::Utils::RateLimits::RateLimiterInterface*, Aws::Utils::RateLimits::RateLimiterInterface*) const in CurlHttpClient.cpp.o
      Aws::Http::CurlHttpClient::MakeRequest(std::__1::shared_ptr<Aws::Http::HttpRequest> const&, Aws::Utils::RateLimits::RateLimiterInterface*, Aws::Utils::RateLimits::RateLimiterInterface*) const in CurlHttpClient.cpp.o
      Aws::Http::CurlHttpClient::MakeRequest(std::__1::shared_ptr<Aws::Http::HttpRequest> const&, Aws::Utils::RateLimits::RateLimiterInterface*, Aws::Utils::RateLimits::RateLimiterInterface*) const in CurlHttpClient.cpp.o
      Aws::Http::CurlHttpClient::MakeRequest(std::__1::shared_ptr<Aws::Http::HttpRequest> const&, Aws::Utils::RateLimits::RateLimiterInterface*, Aws::Utils::RateLimits::RateLimiterInterface*) const in CurlHttpClient.cpp.o
  _curl_slist_free_all, referenced from:
      Aws::Http::CurlHttpClient::MakeRequest(std::__1::shared_ptr<Aws::Http::HttpRequest> const&, Aws::Utils::RateLimits::RateLimiterInterface*, Aws::Utils::RateLimits::RateLimiterInterface*) const in CurlHttpClient.cpp.o
  _curl_version_info, referenced from:
      Aws::Http::CurlHttpClient::InitGlobalState() in CurlHttpClient.cpp.o
clang: error: linker command failed with exit code 1 (use -v to see invocation)
[335/338] Building CXX object generated/src/aws-cpp-sdk-iam/CMakeFiles/aws-cpp-sdk-iam.dir/ub_IAM.cpp.o
ninja: build stopped: subcommand failed.

With my quick hacky change (see #2718 and https://github.com/pr0g/aws-sdk-cpp/tree/experiment-curl-cmake) I am unblocked, but it would be really cool if this could be fixed upstream.

Thanks very much for your time!

pr0g · 2023-10-20T10:07:48Z

FYI this Stack Overflow question/answer seems to describe the same issue (not in the context of aws-sdk-cpp, but it's the same situation described above).

sbiscigl · 2024-01-19T16:21:11Z

Hey @pr0g i opened a PR that optionally falls back to using find_package if we cannot find the missing include and lib directories as mentioned, give it a look and let me know what you think, it should solve your issue as well

pr0g · 2024-01-19T16:51:57Z

Hey @pr0g i opened a PR that optionally falls back to using find_package if we cannot find the missing include and lib directories as mentioned, give it a look and let me know what you think, it should solve your issue as well

Change #2817 looks good to me! Thanks very much for adding support for this! 🥳 🎉

jmklix · 2024-01-24T21:49:06Z

@pr0g with the above PR merged does this work for you? If so we can close this issue

pr0g · 2024-01-24T21:52:59Z

Hey @jmklix, I haven't had a chance to test this out yet but I think it's safe to close for now. When I do get to it, if there's any issues I'll reopen with a repro. Thanks!

github-actions · 2024-01-24T22:49:04Z

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

pr0g added bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels Oct 17, 2023

pr0g mentioned this issue Oct 17, 2023

Experimental change to switch to using find_package and linking against CURL::libcurl target #2718

Closed

11 tasks

jmklix added pending-release This issue will be fixed by an approved PR that hasn't been released yet. p2 This is a standard priority issue and removed needs-triage This issue or PR still needs to be triaged. labels Oct 18, 2023

pr0g mentioned this issue Nov 5, 2023

Linking error #1676

Closed

2 tasks

jmklix added the Cmake Cmake related submissions label Nov 17, 2023

sbiscigl mentioned this issue Jan 19, 2024

use curl in config mode if include and lib missing #2817

Merged

11 tasks

jmklix closed this as completed Jan 24, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CMake Config mode support for Curl #2717

CMake Config mode support for Curl #2717

pr0g commented Oct 17, 2023 •

edited

Loading

SergeyRyabinin commented Oct 17, 2023

pr0g commented Oct 18, 2023

pr0g commented Oct 18, 2023 •

edited

Loading

pr0g commented Oct 20, 2023

sbiscigl commented Jan 19, 2024

pr0g commented Jan 19, 2024

jmklix commented Jan 24, 2024

pr0g commented Jan 24, 2024

github-actions bot commented Jan 24, 2024

CMake Config mode support for Curl #2717

CMake Config mode support for Curl #2717

Comments

pr0g commented Oct 17, 2023 • edited Loading

Describe the bug

Expected Behavior

Current Behavior

Reproduction Steps

Possible Solution

Additional Information/Context

AWS CPP SDK version used

Compiler and Version used

Operating System and version

SergeyRyabinin commented Oct 17, 2023

pr0g commented Oct 18, 2023

pr0g commented Oct 18, 2023 • edited Loading

pr0g commented Oct 20, 2023

sbiscigl commented Jan 19, 2024

pr0g commented Jan 19, 2024

jmklix commented Jan 24, 2024

pr0g commented Jan 24, 2024

github-actions bot commented Jan 24, 2024

⚠️COMMENT VISIBILITY WARNING⚠️

pr0g commented Oct 17, 2023 •

edited

Loading

pr0g commented Oct 18, 2023 •

edited

Loading