tiledb.open("s3://...") errors if pyarrow was imported before tiledb.

[ivirshup]

Repro

• Import order matters here, we don't see this error if tiledb is imported before pyarrow
• This only seems to affect more recent versions of pyarrow

import pyarrow
import tiledb

SOMA_URI= "s3://cellxgene-census-public-us-west-2/cell-census/2023-12-15/soma"
SPECIES = "mus_musculus"

ctx = {
    "vfs.s3.no_sign_request": "true",
    "vfs.s3.region": "us-west-2"
}

tiledb_array = tiledb.open(
    f"{SOMA_URI}/census_data/{SPECIES}/ms/RNA/X/raw/",
    ctx=tiledb.Ctx(ctx),
)

Error:

Fatal error condition occurred in /Users/runner/work/crossbow/crossbow/vcpkg/buildtrees/aws-c-common/src/v0.9.9-5d8eb6bbc6.clean/source/allocator.c:172: allocator != ((void*)0)
Exiting Application
################################################################################
Stack trace:
################################################################################
1   libarrow.1601.dylib                 0x00000001093778d0 aws_fatal_assert + 80
2   libarrow.1601.dylib                 0x0000000109376b98 aws_mem_acquire + 64
3   libarrow.1601.dylib                 0x0000000109389d68 aws_string_new_from_cursor + 76
4   libarrow.1601.dylib                 0x00000001093835e4 aws_json_value_get_from_object + 44
5   libarrow.1601.dylib                 0x000000010936f970 aws_endpoints_ruleset_new_from_string + 120
6   libarrow.1601.dylib                 0x00000001093095a4 _ZN3Aws3Crt9Endpoints10RuleEngineC2ERK15aws_byte_cursorS5_P13aws_allocator + 48
7   libarrow.1601.dylib                 0x00000001090e3180 _ZN3Aws8Endpoint23DefaultEndpointProviderINS_2S321S3ClientConfigurationENS2_8Endpoint19S3BuiltInParametersENS4_25S3ClientContextParametersEEC2EPKcm + 116
8   libtiledb.dylib                     0x00000001235a2308 _ZN3Aws2S38S3ClientC2ERKNSt3__110shared_ptrINS_4Auth22AWSCredentialsProviderEEERKNS_6Client19ClientConfigurationENS9_15AWSAuthV4Signer20PayloadSigningPolicyEbNS0_34US_EAST_1_REGIONAL_ENDPOINT_OPTIONE + 884
9   libtiledb.dylib                     0x0000000122dc35d0 _ZN6tiledb6common11make_sharedINS_2sm14TileDBS3ClientELi66EJRKNS2_12S3ParametersERNSt3__110shared_ptrIN3Aws4Auth22AWSCredentialsProviderEEERNS9_6Client19ClientConfigurationENSE_15AWSAuthV4Signer20PayloadSigningPolicyERKbEEENS8_IT_EERAT0__KcDpOT1_ + 104
10  libtiledb.dylib                     0x0000000122dac628 _ZNK6tiledb2sm2S311init_clientEv + 3384
11  libtiledb.dylib                     0x0000000122db6890 _ZNK6tiledb2sm2S313ls_with_sizesERKNS0_3URIERKNSt3__112basic_stringIcNS5_11char_traitsIcEENS5_9allocatorIcEEEEi + 76
12  libtiledb.dylib                     0x0000000122db67f8 _ZNK6tiledb2sm2S313ls_with_sizesERKNS0_3URIE + 44
13  libtiledb.dylib                     0x0000000122ddc7f4 _ZNK6tiledb2sm3VFS13ls_with_sizesERKNS0_3URIE + 264
14  libtiledb.dylib                     0x0000000122c5b540 _ZNK6tiledb2sm14ArrayDirectory2lsERKNS0_3URIE + 56
15  libtiledb.dylib                     0x0000000122c5e1c8 _ZN6tiledb2sm14ArrayDirectory22load_array_schema_urisEv + 124
16  libtiledb.dylib                     0x0000000122c60784 _ZNSt3__120__packaged_task_funcIZN6tiledb6common10ThreadPool5asyncIZNS1_2sm14ArrayDirectory4loadEvE3$_4JEEEDaOT_DpOT0_EUlvE_NS_9allocatorISE_EEFNS2_6StatusEvEEclEv + 24
17  libtiledb.dylib                     0x0000000123493938 _ZNSt3__113packaged_taskIFN6tiledb6common6StatusEvEEclEv + 76
18  libtiledb.dylib                     0x0000000123493674 _ZN6tiledb6common10ThreadPool6workerEv + 76
19  libtiledb.dylib                     0x0000000123494b74 _ZNSt3__1L14__thread_proxyINS_5tupleIJNS_10unique_ptrINS_15__thread_structENS_14default_deleteIS3_EEEEMN6tiledb6common10ThreadPoolEFvvEPS9_EEEEEPvSE_ + 64
20  libsystem_pthread.dylib             0x0000000195a3af94 _pthread_start + 136
21  libsystem_pthread.dylib             0x0000000195a35d34 thread_start + 8

Environment info:

# packages in environment at /Users/ivirshup/miniforge3/envs/tiledb-pyarrow-bug:
#
# Name                    Version                   Build  Channel
bzip2                     1.0.8                h93a5062_5    conda-forge
ca-certificates           2024.6.2             hf0a4a13_0    conda-forge
libexpat                  2.6.2                hebf3989_0    conda-forge
libffi                    3.4.2                h3422bc3_5    conda-forge
libsqlite                 3.45.3               h091b4b1_0    conda-forge
libzlib                   1.3.1                hfb2fe0b_1    conda-forge
ncurses                   6.5                  hb89a1cb_0    conda-forge
numpy                     1.26.4                   pypi_0    pypi
openssl                   3.3.1                hfb2fe0b_0    conda-forge
packaging                 24.0                     pypi_0    pypi
pip                       24.0               pyhd8ed1ab_0    conda-forge
pyarrow                   16.1.0                   pypi_0    pypi
python                    3.12.3          h4a7b5fc_0_cpython    conda-forge
readline                  8.2                  h92ec313_1    conda-forge
setuptools                70.0.0             pyhd8ed1ab_0    conda-forge
tiledb                    0.29.0                   pypi_0    pypi
tk                        8.6.13               h5083fa2_1    conda-forge
tzdata                    2024a                h0c530f3_0    conda-forge
wheel                     0.43.0             pyhd8ed1ab_1    conda-forge
xz                        5.2.6                h57fd34a_0    conda-forge

See also

This came up while trying to unpin pyarrow (and allow macOS support for python 3.12) on tiledbsoma

• [python] Try 3.12 again [WIP] single-cell-data/TileDB-SOMA#2692

[johnkerl]

We had extensive internal discussions a few months ago and determined that the fault was with pyarrow not tiledb.

I'll resuscitate those discussions freshly. cc @ihnorton.

[ihnorton]

This is fixed in TileDB 2.26 / TileDB-Py 0.32 (and related SOMA versions as well). x-ref TileDB-Inc/TileDB#5223

Thanks for opening the issue.

[johnkerl]

related SOMA versions as well
https://github.com/single-cell-data/TileDB-SOMA/releases/tag/1.14.1