Aws::InitAPI(aws_options_) in two independent libraries may crash on OSX

[levongh]

Describe the bug

I do have a python binding library which is setting up an aws sdk during library init. And in the other hand in my python script, I am also using pyarrow library which crashes with the following stack trace.

Exiting Application
################################################################################
Stack trace:
################################################################################
1   libarrow.1300.dylib                 0x0000000125a09026 aws_fatal_assert + 70
2   libarrow.1300.dylib                 0x0000000125a08214 aws_mem_release + 52
3   libarrow.1300.dylib                 0x0000000125a0e945 cJSON_Delete + 101
4   libarrow.1300.dylib                 0x0000000125a0e92c cJSON_Delete + 76
5   libarrow.1300.dylib                 0x0000000125a00bb7 s_endpoints_ruleset_destroy + 23
6   libarrow.1300.dylib                 0x0000000125a1b1f2 aws_ref_count_release + 34
7   libarrow.1300.dylib                 0x0000000125a00c32 aws_endpoints_ruleset_release + 18
8   libarrow.1300.dylib                 0x00000001259fefd7 s_endpoints_rule_engine_destroy + 23
9   libarrow.1300.dylib                 0x0000000125a1b1f2 aws_ref_count_release + 34
10  libarrow.1300.dylib                 0x00000001259ff042 aws_endpoints_rule_engine_release + 18
11  libarrow.1300.dylib                 0x0000000125994aa1 _ZN3Aws3Crt9Endpoints10RuleEngineD1Ev + 17
12  api.cpython-310-darwin.so           0x000000013b1e4ecc _ZN3Aws2S38S3ClientD0Ev + 108
13  api.cpython-310-darwin.so           0x000000013aaafd86 api.cpython-310-darwin.so + 11033990
14  api.cpython-310-darwin.so           0x000000013aaafe2e api.cpython-310-darwin.so + 11034158
15  api.cpython-310-darwin.so           0x000000013a1bafd6 api.cpython-310-darwin.so + 1642454
16  api.cpython-310-darwin.so           0x000000013a140855 api.cpython-310-darwin.so + 1140821
17  api.cpython-310-darwin.so           0x000000013b7517d2 _ZN8pybind1117error_already_set23m_fetched_error_deleterEPNS_6detail25error_fetch_and_normalizeE + 64066
18  api.cpython-310-darwin.so           0x000000013aa6e39a api.cpython-310-darwin.so + 10765210
19  api.cpython-310-darwin.so           0x000000013aa6e23b api.cpython-310-darwin.so + 10764859
20  libsystem_pthread.dylib             0x00007ff81f5e44e1 _pthread_start + 125
21  libsystem_pthread.dylib             0x00007ff81f5dff6b thread_start + 15

Expected Behavior

The expected behavior is to run and keep two independent libraries to work together without contradicting each other's state.

Current Behavior

it crashes

Reproduction Steps

#include <pybind11/pybind11.h>

class A{

public:
    A();
}

A::A()
{
     Aws::SDKOptions aws_options;
     Aws::InitAPI(aws_options)
}

A::~A()
{
     Aws::SDKOptions aws_options;
     Aws::ShutdownAPI(aws_options)
}


int add(int i, int j) {
    //create aws s3 client and make calls
    return i + j;
}

namespace py = pybind11;

PYBIND11_MODULE(python_example, m) {
   static A instance;  // Aws sdk is initialized in the constructor
    m.def("add", &A::add);
}

python3 -m pip install pyarrow.
python3
>>> import pyarrow
>>> import python_example
>>> python_example.add(1, 2)

Exiting Application
################################################################################
Stack trace:
################################################################################
1   libarrow.1300.dylib                 0x0000000126639f7c aws_fatal_assert + 80
2   libarrow.1300.dylib                 0x00000001266393dc aws_mem_acquire + 64
3   libarrow.1300.dylib                 0x000000012664b024 aws_string_new_from_cursor + 60
4   libarrow.1300.dylib                 0x0000000126645510 aws_json_value_get_from_object + 44
5   libarrow.1300.dylib                 0x0000000126632484 aws_endpoints_ruleset_new_from_string + 120
6   libarrow.1300.dylib                 0x00000001265ce488 _ZN3Aws3Crt9Endpoints10RuleEngineC2ERK15aws_byte_cursorS5_P13aws_allocator + 48
7   libarrow.1300.dylib                 0x0000000125d77938 _ZN3Aws8Endpoint23DefaultEndpointProviderINS_2S321S3ClientConfigurationENS2_8Endpoint19S3BuiltInParametersENS4_25S3ClientContextParametersEEC2EPKcm + 120
8   api.cpython-311-darwin.so           0x000000015cec3028 _ZN3Aws2S38S3ClientC2ERKNSt3__110shared_ptrINS_4Auth22AWSCredentialsProviderEEERKNS_6Client19ClientConfigurationENS9_15AWSAuthV4Signer20PayloadSigningPolicyEbNS0_34US_EAST_1_REGIONAL_ENDPOINT_OPTIONE + 440
9   api.cpython-311-darwin.so           0x000000015c9a6348 api.cpython-311-darwin.so + 10117960
10  api.cpython-311-darwin.so           0x000000015c9a5584 api.cpython-311-darwin.so + 10114436
11  api.cpython-311-darwin.so           0x000000015d49217c PyInit_api + 211588
12  api.cpython-311-darwin.so           0x000000015d493330 PyInit_api + 216120
13  api.cpython-311-darwin.so           0x000000015d434fd4 _ZN8pybind1117error_already_set23m_fetched_error_deleterEPNS_6detail25error_fetch_and_normalizeE + 38828
14  api.cpython-311-darwin.so           0x000000015d447ffc _ZN8pybind1117error_already_set23m_fetched_error_deleterEPNS_6detail25error_fetch_and_normalizeE + 116692
15  api.cpython-311-darwin.so           0x000000015d426850 _ZN6crc32c11ExtendArm64EjPKhm + 50052
16  Python                              0x0000000102de6980 cfunction_call + 60
17  Python                              0x0000000102d9eeb8 _PyObject_MakeTpCall + 128
18  Python                              0x0000000102da2440 method_vectorcall + 564
19  Python                              0x0000000102d9f984 _PyVectorcall_Call + 116
20  Python                              0x0000000102e04d6c slot_tp_init + 464
21  Python                              0x0000000102dfd5d0 type_call + 144
22  libtorch_python.dylib               0x000000011e539f28 pybind11_meta_call + 40
23  Python                              0x0000000102d9eeb8 _PyObject_MakeTpCall + 128
24  Python                              0x0000000102e7715c _PyEval_EvalFrameDefault + 40660
25  Python                              0x0000000102e6c744 PyEval_EvalCode + 168
26  Python                              0x0000000102ebdf6c run_eval_code_obj + 84
27  Python                              0x0000000102ebded0 run_mod + 112
28  Python                              0x0000000102ebe204 PyRun_InteractiveOneObjectEx + 548
29  Python                              0x0000000102ebd5c8 _PyRun_InteractiveLoopObject + 288
30  Python                              0x0000000102ebd068 _PyRun_AnyFileObject + 76
31  Python                              0x0000000102ebff94 PyRun_AnyFileExFlags + 84
32  Python                              0x0000000102ed8a04 pymain_run_stdin + 160
33  Python                              0x0000000102ed8050 Py_RunMain + 348
34  Python                              0x0000000102ed92ec Py_BytesMain + 40
35  dyld                                0x0000000197b0ff28 start + 2236

Possible Solution

No response

Additional Information/Context

No response

AWS CPP SDK version used

1.10.55

Compiler and Version used

clang 13

Operating System and version

OSX

[jmklix]

Currently you can only call InitAPI and ShutdownAPI once per application. If you want to use pyarrow you would need to make sure that InitAPI and ShutdownAPI are only called once. I'm not super familiar with that library, but you would need to control the init of the aws-sdk-cpp yourself and have pyarrow not initialize this sdk. That is something you you can try to do right now.

Ideally this sdk would have a reference counter that would handle situations like what you are trying to do above. I will be switching this bug to a feature request. This is not something that we currently have planned, but it is something that we would like to do. I will make sure that we update this ticket with any progress.

[jmklix]

We have a feature request already open for this same issue. Closing this one
#2411

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.