Fatal error with pyarrow/libarrow.so

[catalys1]

Describe the bug

When using datasets, at the very end of my jobs the program crashes (see trace below).
It doesn't seem to affect anything, as it appears to happen as the program is closing down. Just importing datasets is enough to cause the error.

Steps to reproduce the bug

This is sufficient to reproduce the problem:

python -c "import datasets"

Expected results

Program should run to completion without an error.

Actual results

Fatal error condition occurred in /opt/vcpkg/buildtrees/aws-c-io/src/9e6648842a-364b708815.clean/source/event_loop.c:72: aws_thread_launch(&cleanup_thread, s_event_loop_destroy_async_thread_fn, el_group, &thread_options) == AWS_OP_SUCCESS
Exiting Application
################################################################################
Stack trace:
################################################################################
/u/user/miniconda3/envs/env/lib/python3.10/site-packages/pyarrow/libarrow.so.900(+0x200af06) [0x150dff547f06]
/u/user/miniconda3/envs/env/lib/python3.10/site-packages/pyarrow/libarrow.so.900(+0x20028e5) [0x150dff53f8e5]
/u/user/miniconda3/envs/env/lib/python3.10/site-packages/pyarrow/libarrow.so.900(+0x1f27e09) [0x150dff464e09]
/u/user/miniconda3/envs/env/lib/python3.10/site-packages/pyarrow/libarrow.so.900(+0x200ba3d) [0x150dff548a3d]
/u/user/miniconda3/envs/env/lib/python3.10/site-packages/pyarrow/libarrow.so.900(+0x1f25948) [0x150dff462948]
/u/user/miniconda3/envs/env/lib/python3.10/site-packages/pyarrow/libarrow.so.900(+0x200ba3d) [0x150dff548a3d]
/u/user/miniconda3/envs/env/lib/python3.10/site-packages/pyarrow/libarrow.so.900(+0x1ee0b46) [0x150dff41db46]
/u/user/miniconda3/envs/env/lib/python3.10/site-packages/pyarrow/libarrow.so.900(+0x194546a) [0x150dfee8246a]
/lib64/libc.so.6(+0x39b0c) [0x150e15eadb0c]
/lib64/libc.so.6(on_exit+0) [0x150e15eadc40]
/u/user/miniconda3/envs/env/bin/python(+0x28db18) [0x560ae370eb18]
/u/user/miniconda3/envs/env/bin/python(+0x28db4b) [0x560ae370eb4b]
/u/user/miniconda3/envs/env/bin/python(+0x28db90) [0x560ae370eb90]
/u/user/miniconda3/envs/env/bin/python(_PyRun_SimpleFileObject+0x1e6) [0x560ae37123e6]
/u/user/miniconda3/envs/env/bin/python(_PyRun_AnyFileObject+0x44) [0x560ae37124c4]
/u/user/miniconda3/envs/env/bin/python(Py_RunMain+0x35d) [0x560ae37135bd]
/u/user/miniconda3/envs/env/bin/python(Py_BytesMain+0x39) [0x560ae37137d9]
/lib64/libc.so.6(__libc_start_main+0xf3) [0x150e15e97493]
/u/user/miniconda3/envs/env/bin/python(+0x2125d4) [0x560ae36935d4]
Aborted (core dumped)

Environment info

• datasets version: 2.5.1
• Platform: Linux-4.18.0-348.23.1.el8_5.x86_64-x86_64-with-glibc2.28
• Python version: 3.10.4
• PyArrow version: 9.0.0
• Pandas version: 1.4.3

[albertvillanova]

Thanks for reporting, @catalys1.

This seems a duplicate of:

• Fatal error condition occurred in aws-c-io #3310

The source of the problem is in PyArrow:

• ARROW-15141: [C++] Fatal error condition occurred in aws_thread_launch
• ARROW-17501: [C++] Fatal error condition occurred in aws_thread_launch

The bug in their dependency is still unresolved:

• Aws::Crt::Io::ClientBootstrap destructor may launch thread at process exit, and crash aws/aws-sdk-cpp#1809

Apparently, the aws-sdk-cpp PyArrow dependency needs to be pinned at version 1.8.186 if using conda. Have you updated it after installing PyArrow?

conda list aws-sdk-cpp

Maybe you should try to downgrade it to that version:

conda install -c conda-forge aws-sdk-cpp=1.8.186