[Python] Flaky Segfault on Python Interpreter Exit

[jayceslesar]

Describe the bug, including details regarding any error messages, version, and platform.

Not sure if I can provide a reproducible minimal example but I can for sure say that arrow is causing a segfault when the python interpreter is exiting, after all the code has finished executing. Below is all the arrow related tracebacks I could get using pystack when analyzing a core dump from a case where there was a segfault on interpreter exit.

(pystack-env) ubuntu@ip-10-253-142-125:~/jayce$ pystack core /var/lib/apport/coredump/core._usr_bin_python3_10.1000.0a6b58e9-a636-4fd7-abcd-62e73f640aba.47132.50099076 /home/ubuntu/jayce/my-project-using-pyarrow/.venv/bin/python --native-all

Core file information:
state: R zombie: True niceness: 0
pid: 47132 ppid: 43292 sid: 40915
uid: 1000 gid: 1000 pgrp: 43285
executable: project-using-pyarrow arguments: /home/ubuntu/jayce/my-project-using-pyarrow/.venv/bin/python /home/ubuntu/jayce/py

The process died due a segmentation fault accessing address: 0x10
Traceback for thread 47132 [] (most recent call last):
    (C) File "../Modules/_statisticsmodule.c", line 91, in _start (python)
    (C) File "../csu/libc-start.c", line 392, in __libc_start_main@@GLIBC_2.34 (libc.so.6)
    (C) File "../sysdeps/nptl/libc_start_call_main.h", line 58, in __libc_start_call_main (libc.so.6)
    (C) File "../Modules/main.c", line 720, in Py_BytesMain (python)
    (C) File "../Modules/main.c", line 353, in Py_RunMain (python)
    (C) File "../Python/pythonrun.c", line 785, in PyErr_PrintEx (python)
    (C) File "../Python/pythonrun.c", line 775, in handle_system_exit.lto_priv.0 (python)
    (C) File "./stdlib/exit.c", line 143, in on_exit (libc.so.6)
    (C) File "./stdlib/exit.c", line 113, in __run_exit_handlers (libc.so.6)
    (C) File "???", line 0, in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() (libarrow.so.1200)
    (C) File "???", line 0, in std::_Sp_counted_ptr<arrow::fs::(anonymous namespace)::RegionResolver*, (__gnu_cxx::_Lock_policy)2>::_M_dispose() (libarrow.so.1200)
    (C) File "???", line 0, in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() (libarrow.so.1200)
    (C) File "???", line 0, in Aws::S3::S3Client::~S3Client() (libarrow.so.1200)
    (C) File "???", line 0, in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() (libarrow.so.1200)
    (C) File "???", line 0, in Aws::Http::CurlHttpClient::~CurlHttpClient() (libarrow.so.1200)
    (C) File "???", line 0, in Aws::Http::CurlHandleContainer::~CurlHandleContainer() (libarrow.so.1200)

Traceback for thread 47237 [] (most recent call last):
    (C) File "../sysdeps/unix/sysv/linux/x86_64/clone3.S", line 81, in __clone3 (libc.so.6)
    (C) File "./nptl/pthread_create.c", line 442, in start_thread (libc.so.6)
    (C) File "src/background_thread.c", line 444, in background_thread_entry (libarrow.so.1200)
    (C) File "src/background_thread.c", line 419, in background_work (inlined) (libarrow.so.1200)
    (C) File "src/background_thread.c", line 229, in background_work_sleep_once (inlined) (libarrow.so.1200)
    (C) File "src/background_thread.c", line 159, in background_thread_sleep (inlined) (libarrow.so.1200)
    (C) File "./nptl/pthread_cond_wait.c", line 652, in pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6)
    (C) File "./nptl/pthread_cond_wait.c", line 503, in __pthread_cond_wait_common (inlined) (libc.so.6)
    (C) File "./nptl/futex-internal.c", line 139, in __GI___futex_abstimed_wait_cancelable64 (libc.so.6)
    (C) File "./nptl/futex-internal.c", line 87, in __futex_abstimed_wait_common (inlined) (libc.so.6)
    (C) File "./nptl/futex-internal.c", line 57, in __futex_abstimed_wait_common64 (inlined) (libc.so.6)

Traceback for thread 47176 [] (most recent call last):
    (C) File "../sysdeps/unix/sysv/linux/x86_64/clone3.S", line 81, in __clone3 (libc.so.6)
    (C) File "./nptl/pthread_create.c", line 442, in start_thread (libc.so.6)
    (C) File "???", line 0, in thread_fn (libarrow.so.1200)
    (C) File "???", line 0, in aws_event_loop_thread (libarrow.so.1200)
    (C) File "../sysdeps/unix/sysv/linux/epoll_wait.c", line 30, in epoll_wait (libc.so.6)

Traceback for thread 47164 [] (most recent call last):
    (C) File "../sysdeps/unix/sysv/linux/x86_64/clone3.S", line 81, in __clone3 (libc.so.6)
    (C) File "./nptl/pthread_create.c", line 442, in start_thread (libc.so.6)
    (C) File "src/background_thread.c", line 444, in background_thread_entry (libarrow.so.1200)
    (C) File "src/background_thread.c", line 412, in background_work (inlined) (libarrow.so.1200)
    (C) File "src/background_thread.c", line 374, in background_thread0_work (inlined) (libarrow.so.1200)
    (C) File "src/background_thread.c", line 229, in background_work_sleep_once (inlined) (libarrow.so.1200)
    (C) File "src/background_thread.c", line 159, in background_thread_sleep (inlined) (libarrow.so.1200)
    (C) File "./nptl/pthread_cond_wait.c", line 652, in pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6)
    (C) File "./nptl/pthread_cond_wait.c", line 503, in __pthread_cond_wait_common (inlined) (libc.so.6)
    (C) File "./nptl/futex-internal.c", line 139, in __GI___futex_abstimed_wait_cancelable64 (libc.so.6)
    (C) File "./nptl/futex-internal.c", line 87, in __futex_abstimed_wait_common (inlined) (libc.so.6)
    (C) File "./nptl/futex-internal.c", line 57, in __futex_abstimed_wait_common64 (inlined) (libc.so.6)

Traceback for thread 47173 [] (most recent call last):
    (C) File "../sysdeps/unix/sysv/linux/x86_64/clone3.S", line 81, in __clone3 (libc.so.6)
    (C) File "./nptl/pthread_create.c", line 442, in start_thread (libc.so.6)
    (C) File "???", line 0, in thread_fn (libarrow.so.1200)
    (C) File "???", line 0, in aws_event_loop_thread (libarrow.so.1200)
    (C) File "../sysdeps/unix/sysv/linux/epoll_wait.c", line 30, in epoll_wait (libc.so.6)

Traceback for thread 47245 [] (most recent call last):
    (C) File "../sysdeps/unix/sysv/linux/x86_64/clone3.S", line 81, in __clone3 (libc.so.6)
    (C) File "./nptl/pthread_create.c", line 442, in start_thread (libc.so.6)
    (C) File "???", line 0, in thread_fn (libarrow.so.1200)
    (C) File "???", line 0, in s_event_loop_destroy_async_thread_fn (libarrow.so.1200)
    (C) File "???", line 0, in s_aws_event_loop_group_shutdown_sync (libarrow.so.1200)
    (C) File "???", line 0, in s_destroy (libarrow.so.1200)
    (C) File "???", line 0, in aws_thread_join (libarrow.so.1200)
    (C) File "./nptl/pthread_join_common.c", line 105, in __pthread_clockjoin_ex (libc.so.6)
    (C) File "./nptl/futex-internal.c", line 139, in __GI___futex_abstimed_wait_cancelable64 (libc.so.6)
    (C) File "./nptl/futex-internal.c", line 87, in __futex_abstimed_wait_common (inlined) (libc.so.6)
    (C) File "./nptl/futex-internal.c", line 57, in __futex_abstimed_wait_common64 (inlined) (libc.so.6)

Traceback for thread 47175 [] (most recent call last):
    (C) File "../sysdeps/unix/sysv/linux/x86_64/clone3.S", line 81, in __clone3 (libc.so.6)
    (C) File "./nptl/pthread_create.c", line 442, in start_thread (libc.so.6)
    (C) File "???", line 0, in thread_fn (libarrow.so.1200)
    (C) File "???", line 0, in aws_event_loop_thread (libarrow.so.1200)
    (C) File "../sysdeps/unix/sysv/linux/epoll_wait.c", line 30, in epoll_wait (libc.so.6)

Traceback for thread 47174 [] (most recent call last):
    (C) File "../sysdeps/unix/sysv/linux/x86_64/clone3.S", line 81, in __clone3 (libc.so.6)
    (C) File "./nptl/pthread_create.c", line 442, in start_thread (libc.so.6)
    (C) File "???", line 0, in thread_fn (libarrow.so.1200)
    (C) File "???", line 0, in aws_event_loop_thread (libarrow.so.1200)
    (C) File "../sysdeps/unix/sysv/linux/epoll_wait.c", line 30, in epoll_wait (libc.so.6)

Traceback for thread 47177 [] (most recent call last):
    (C) File "../sysdeps/unix/sysv/linux/x86_64/clone3.S", line 81, in __clone3 (libc.so.6)
    (C) File "./nptl/pthread_create.c", line 442, in start_thread (libc.so.6)
    (C) File "???", line 0, in thread_fn (libarrow.so.1200)
    (C) File "???", line 0, in aws_event_loop_thread (libarrow.so.1200)
    (C) File "../sysdeps/unix/sysv/linux/epoll_wait.c", line 30, in epoll_wait (libc.so.6)

Traceback for thread 47238 [] (most recent call last):
    (C) File "../sysdeps/unix/sysv/linux/x86_64/clone3.S", line 81, in __clone3 (libc.so.6)
    (C) File "./nptl/pthread_create.c", line 442, in start_thread (libc.so.6)
    (C) File "src/background_thread.c", line 444, in background_thread_entry (libarrow.so.1200)
    (C) File "src/background_thread.c", line 419, in background_work (inlined) (libarrow.so.1200)
    (C) File "src/background_thread.c", line 229, in background_work_sleep_once (inlined) (libarrow.so.1200)
    (C) File "src/background_thread.c", line 159, in background_thread_sleep (inlined) (libarrow.so.1200)
    (C) File "./nptl/pthread_cond_wait.c", line 652, in pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6)
    (C) File "./nptl/pthread_cond_wait.c", line 503, in __pthread_cond_wait_common (inlined) (libc.so.6)
    (C) File "./nptl/futex-internal.c", line 139, in __GI___futex_abstimed_wait_cancelable64 (libc.so.6)
    (C) File "./nptl/futex-internal.c", line 87, in __futex_abstimed_wait_common (inlined) (libc.so.6)
    (C) File "./nptl/futex-internal.c", line 57, in __futex_abstimed_wait_common64 (inlined) (libc.so.6)

Traceback for thread 47235 [] (most recent call last):
    (C) File "../sysdeps/unix/sysv/linux/x86_64/clone3.S", line 81, in __clone3 (libc.so.6)
    (C) File "./nptl/pthread_create.c", line 442, in start_thread (libc.so.6)
    (C) File "src/background_thread.c", line 444, in background_thread_entry (libarrow.so.1200)
    (C) File "src/background_thread.c", line 419, in background_work (inlined) (libarrow.so.1200)
    (C) File "src/background_thread.c", line 229, in background_work_sleep_once (inlined) (libarrow.so.1200)
    (C) File "src/background_thread.c", line 159, in background_thread_sleep (inlined) (libarrow.so.1200)
    (C) File "./nptl/pthread_cond_wait.c", line 652, in pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6)
    (C) File "./nptl/pthread_cond_wait.c", line 503, in __pthread_cond_wait_common (inlined) (libc.so.6)
    (C) File "./nptl/futex-internal.c", line 139, in __GI___futex_abstimed_wait_cancelable64 (libc.so.6)
    (C) File "./nptl/futex-internal.c", line 87, in __futex_abstimed_wait_common (inlined) (libc.so.6)
    (C) File "./nptl/futex-internal.c", line 57, in __futex_abstimed_wait_common64 (inlined) (libc.so.6)

Component(s)

C++

[jayceslesar]

looks related to #38156 so I will test that and come back with results

[jayceslesar]

closing as this is not an issue in >= 13.0.0

[jorisvandenbossche]

This might have been fixed by #15054 / #33858