Skip to content

Any query will crash searchd while adding data #133

@cappadaan

Description

@cappadaan

Bug Description:

Manticore 17.5

  1. Create a table with vector

vector FLOAT_VECTOR knn_type='hnsw' hnsw_similarity='cosine' model_name='Qwen/Qwen3-Embedding-0.6B' from='ai_content'

  1. Run a script which insert data

  2. While the script is running, any select on the index crashes searchd

  3. Disable the script, any select on the index does not crash searchd

Crash log

------- FATAL: CRASH DUMP -------
[Sat Feb 7 10:57:58.090 2026] [1963935]

--- crashed SphinxQL request dump ---
SELECT * FROM xxx WHERE knn(vector, 5, 'query', { ef=300, oversampling=3.0,
rescore=1 })
--- request dump end ---
--- local index:xxx
Manticore 17.5.1 d4cc0969e@26020616 (columnar 10.2.0 9a5dd2f@26020506) (secondary 10.2.0 9a5dd2f@26020506) (knn 10.2.0 9a5dd2f@26020506) (embeddings 1.1.1)
Handling signal 6
-------------- backtrace begins here ---------------
Program compiled with Clang 16.0.6
Configured with flags: Configured with these definitions: -DDISTR_BUILD=rhel9 -DUSE_SYSLOG=1 -DWITH_GALERA=1 -DWITH_RE2=1 -DWITH_RE2_FORCE_STATIC=1 -DWITH_STEMMER=1 -DWITH_STEMMER_FORCE_STATIC=1 -DWITH_NLJSON=1 -DWITH_UNIALGO=1 -DWITH_ICU=1 -DWITH_ICU_FORCE_STATIC=1 -DWITH_JIEBA=1 -DWITH_SSL=1 -DWITH_ZLIB=1 -DWITH_ZSTD=1 -DDL_ZSTD=1 -DZSTD_LIB=libzstd.so.1 -DWITH_CURL=1 -DDL_CURL=1 -DCURL_LIB=libcurl.so.4 -DWITH_ODBC=1 -DDL_ODBC=1 -DODBC_LIB=libodbc.so.2 -DWITH_EXPAT=1 -DDL_EXPAT=1 -DEXPAT_LIB=libexpat.so.1 -DWITH_ICONV=1 -DWITH_MYSQL=1 -DDL_MYSQL=1 -DMYSQL_LIB=libmariadb.so.3 -DWITH_POSTGRESQL=1 -DDL_POSTGRESQL=1 -DPOSTGRESQL_LIB=libpq.so.5 -DLOCALDATADIR=/var/lib/manticore -DFULL_SHARE_DIR=/usr/share/manticore
Built on Linux x86_64 (rhel9) (cross-compiled)
Stack bottom = 0x7f7bcc063fb0, thread stack size = 0x20000
Trying manual backtrace:
Something wrong with thread stack, manual backtrace may be incorrect (fp=0x20000)
Wrong stack limit or frame pointer, manual backtrace failed (fp=0x20000, stack=0x7f7bcc060000, stacksize=0x20000)
Trying system backtrace:
begin of system symbols:
searchd(_Z12sphBacktraceib+0x227)[0x55ae77cc4da7]
searchd(_Z11HandleCrashi+0x2e3)[0x55ae779d5253]
/lib64/libc.so.6(+0x3fc30)[0x7f82ae43fc30]
/lib64/libc.so.6(+0x8d02c)[0x7f82ae48d02c]
/lib64/libc.so.6(raise+0x16)[0x7f82ae43fb86]
/lib64/libc.so.6(abort+0xd3)[0x7f82ae429873]
/usr/share/manticore/modules/lib_manticore_knn_embeddings.so(+0x53ee0a)[0x7f82ad73ee0a]
/usr/share/manticore/modules/lib_manticore_knn_embeddings.so(+0x518be9)[0x7f82ad718be9]
/usr/share/manticore/modules/lib_manticore_knn_embeddings.so(+0x518c8d)[0x7f82ad718c8d]
/usr/share/manticore/modules/lib_manticore_knn_embeddings.so(+0x518150)[0x7f82ad718150]
/usr/share/manticore/modules/lib_manticore_knn_embeddings.so(+0x53f2c8)[0x7f82ad73f2c8]
/usr/share/manticore/modules/lib_manticore_knn_embeddings.so(+0x53f229)[0x7f82ad73f229]
/usr/share/manticore/modules/lib_manticore_knn_embeddings.so(+0x53f21c)[0x7f82ad73f21c]
/usr/share/manticore/modules/lib_manticore_knn_embeddings.so(+0x39218f)[0x7f82ad59218f]
/usr/share/manticore/modules/lib_manticore_knn_embeddings.so(+0x39b3b2)[0x7f82ad59b3b2]
/usr/share/manticore/modules/lib_manticore_knn_embeddings.so(+0x39b359)[0x7f82ad59b359]
/usr/share/manticore/modules/lib_manticore_knn_embeddings.so(+0x28a1e3)[0x7f82ad48a1e3]
/usr/share/manticore/modules/lib_manticore_knn_embeddings.so(+0x297b79)[0x7f82ad497b79]

/usr/share/manticore/modules/lib_manticore_knn_avx2.so(_ZNK3knn18TextToEmbeddings_c7ConvertERKSt6vectorISt17basic_string_viewIcSt11char_traitsIcEESaIS5_EERS1_IS1_IfSaIfEESaISB_EERNSt7__cxx1112basic_stringIcS4_SaIcEEE+0x16b)[0x7f82ae37d07b]
searchd(_ZNK9RtIndex_c19SetupAutoEmbeddingsERK9CSphQueryRS0_RK10ISphSchemaR10CSphString+0x106)[0x55ae77dcc9b6]
searchd(_ZNK9RtIndex_c10MultiQueryER15CSphQueryResultRK9CSphQueryRK11VecTraits_TIP15ISphMatchSorterERK18CSphMultiQueryArgs+0x6fc)[0x55ae77dcd33c]
searchd(_ZNK13CSphIndexStub12MultiQueryExEiPK9CSphQueryP15CSphQueryResultPP15ISphMatchSorterRK18CSphMultiQueryArgs+0x70)[0x55ae77b0edd0]
searchd(+0x1033b70)[0x55ae779f7b70]
searchd(+0x153fe7d)[0x55ae77f03e7d]
searchd(_ZN7Threads4Coro8ExecuteNEiOSt8functionIFvvEE+0x7a)[0x55ae77bb73aa]
searchd(_ZN15SearchHandler_c16RunLocalSearchesEv+0x77d)[0x55ae779ebccd]
searchd(_ZN15SearchHandler_c9RunSubsetEii+0x51c)[0x55ae779ed00c]
searchd(_ZN15SearchHandler_c10RunQueriesEv+0xa8)[0x55ae779eca78]
searchd(_Z17HandleMysqlSelectR11RowBuffer_iR15SearchHandler_c+0x158)[0x55ae77b44608]
searchd(_ZN15ClientSession_c7ExecuteESt4pairIPKciER11RowBuffer_i+0x1b4d)[0x55ae77b53c3d]
searchd(_Z8SqlServeSt10unique_ptrI16AsyncNetBuffer_cSt14default_deleteIS0_EE+0x1b6b)[0x55ae77a73afb]
searchd(_Z10MultiServeSt10unique_ptrI16AsyncNetBuffer_cSt14default_deleteIS0_EESt4pairIitE7Proto_e+0x43)[0x55ae77a6e6e3]
searchd(+0x10ab28f)[0x55ae77a6f28f]
searchd(ZZN7Threads11CoRoutine_c13CreateContextESt8functionIFvvEESt4pairIN5boost7context13stack_contextENS_14StackFlavour_EEEENUlNS6_6detail10transfer_tEE_8__invokeESB+0x1c)[0x55ae77bba38c]
searchd(make_fcontext+0x2f)[0x55ae7916708f]
Trying boost backtrace:
0# sphBacktrace(int, bool) in searchd
1# HandleCrash(int) in searchd
2# 0x00007F82AE43FC30 in /lib64/libc.so.6
3# 0x00007F82AE48D02C in /lib64/libc.so.6
4# raise in /lib64/libc.so.6
5# abort in /lib64/libc.so.6
6# 0x00007F82AD73EE0A in /usr/share/manticore/modules/lib_manticore_knn_embeddings.so
7# 0x00007F82AD718BE9 in /usr/share/manticore/modules/lib_manticore_knn_embeddings.so
8# 0x00007F82AD718C8D in /usr/share/manticore/modules/lib_manticore_knn_embeddings.so
9# 0x00007F82AD718150 in /usr/share/manticore/modules/lib_manticore_knn_embeddings.so
10# 0x00007F82AD73F2C8 in /usr/share/manticore/modules/lib_manticore_knn_embeddings.so
11# 0x00007F82AD73F229 in /usr/share/manticore/modules/lib_manticore_knn_embeddings.so
12# 0x00007F82AD73F21C in /usr/share/manticore/modules/lib_manticore_knn_embeddings.so
13# 0x00007F82AD59218F in /usr/share/manticore/modules/lib_manticore_knn_embeddings.so
14# 0x00007F82AD59B3B2 in /usr/share/manticore/modules/lib_manticore_knn_embeddings.so
15# 0x00007F82AD59B359 in /usr/share/manticore/modules/lib_manticore_knn_embeddings.so
16# 0x00007F82AD48A1E3 in /usr/share/manticore/modules/lib_manticore_knn_embeddings.so
17# 0x00007F82AD497B79 in /usr/share/manticore/modules/lib_manticore_knn_embeddings.so
18# knn::TextToEmbeddings_c::Convert(std::vector<std::basic_string_view<char, std::char_traits >, std::allocator<std::basic_string_view<char, std::char_traits > > > const&, std::vector<std::vector<float, std::allocator >, std::allocator<std::vector<float, std::allocator > > >&, std::__cxx11::basic_string<char, std::char_traits, std::allocator >&) const in /usr/share/manticore/modules/lib_manticore_knn_avx2.so
19# RtIndex_c::SetupAutoEmbeddings(CSphQuery const&, CSphQuery&, ISphSchema const&, CSphString&) const in searchd
20# RtIndex_c::MultiQuery(CSphQueryResult&, CSphQuery const&, VecTraits_T<ISphMatchSorter*> const&, CSphMultiQueryArgs const&) const in searchd
21# CSphIndexStub::MultiQueryEx(int, C
23# 0x000055AE77F03E7D in searchd
24# Threads::Coro::ExecuteN(int, std::function<void ()>&&) in searchd
25# SearchHandler_c::RunLocalSearches() in searchd
26# SearchHandler_c::RunSubset(int, int) in searchd
27# SearchHandler_c::RunQueries() in searchd
28# HandleMysqlSelect(RowBuffer_i&, SearchHandler_c&) in searchd
29# ClientSession_c::Execute(std::pair<char const*, int>, RowBuffer_i&) in searchd
30# SqlServe(std::unique_ptr<AsyncNetBuffer_c, std::default_delete<AsyncNetBuffer_c> >) in searchd
31# MultiServe(std::unique_ptr<AsyncNetBuffer_c, std::default_delete<AsyncNetBuffer_c> >, std::pair<int, unsigned short>, Proto_e) in searchd
32# 0x000055AE77A6F28F in searchd
33# Threads::CoRoutine_c::CreateContext(std::function<void ()>, std::pair<boost::context::stack_context, Threads::StackFlavour_E>)::{lambda(boost::context::detail::transfer_t)#1}::__invoke(boost::context::detail::transfer_t) in searchd
34# make_fcontext in searchd

-------------- backtrace ends here ---------------
Please, create a bug report in our bug tracker (https://github.com/manticoresoftware/manticore/issues)
and attach there:
a) searchd log, b) searchd binary, c) searchd symbols.
Look into the chapter 'Reporting bugs' in the manual
(https://manual.manticoresearch.com/Reporting_bugs)
Dump with GDB via watchdog
--- active threads ---
thd 0 (work_0), proto mysql, state query, command select
thd 1 (work_3), proto http, state query, command replace
--- Totally 3 threads, and 2 client-working threads ---
------- CRASH DUMP END -------

Manticore Search Version:

17.5

Operating System Version:

Almalinux

Have you tried the latest development version?

None

Internal Checklist:

To be completed by the assignee. Check off tasks that have been completed or are not applicable.

Details
  • Implementation completed
  • Tests developed
  • Documentation updated
  • Documentation reviewed

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions