Can't start manticore after update from 5.0.2 to 6.3.8, OOM error

[dzek69]

Bug Description:

I use docker images, I used 5.0.2 and decided to upgrade. After upgrading the image to 6.3.8 Manticore crashes with OOM error:
FATAL: out of memory (unable to allocate 14401991088 bytes)

I have 8GB of RAM, Manticore tries to allocate ~14GB in memory (during loading not-so-big table, I have much bigger tables that used to work just fine).

I reported this issue first on Telegram: https://t.me/manticoresearch_en/1678 and apparently if you have enough RAM Manticore won't crash, but won't finish starting anyway.

I uploaded my tables with issue-20241201 name.

Sergey on Telegram suggested this seems to be a data backward compatibility issue.

Manticore Search Version:

daemon version '6.3.8 d17bd2b6b@24112202 (columnar 2.3.0 88a01c3@24052206) (secondary 2.3.0 88a01c3@24052206) (knn 2.3.0 88a01c3@24052206)'

Operating System Version:

Debian GNU/Linux 12 (bookworm)

Have you tried the latest development version?

No

Internal Checklist:

^{To be completed by the assignee. Check off tasks that have been completed or are not applicable.}

Details

• Implementation completed

[sanikolaev]

I could reproduce the issue so:

# docker run -v $(pwd)/data:/var/lib/manticore --rm --name=manticore manticoresearch/manticore:7.4.6
[Sat Mar 22 08:41:04.452 2025] [1] using config file '/etc/manticoresearch/manticore.conf.sh' (353 chars)...
starting daemon version '7.4.6 b2ff82920@25022808 (columnar 4.1.1 25f4706@25022806) (secondary 4.1.1 25f4706@25022806) (knn 4.1.1 25f4706@25022806)' ...
listening on all interfaces for mysql, port=9306
listening on UNIX socket /var/run/mysqld/mysqld.sock
listening on all interfaces for sphinx and http(s), port=9308
listening on 172.17.0.12:9312 for sphinx and http(s)
Manticore 7.4.6 b2ff82920@25022808 (columnar 4.1.1 25f4706@25022806) (secondary 4.1.1 25f4706@25022806) (knn 4.1.1 25f4706@25022806)
Copyright (c) 2001-2016, Andrew Aksyonoff
Copyright (c) 2008-2016, Sphinx Technologies Inc (http://sphinxsearch.com)
Copyright (c) 2017-2024, Manticore Software LTD (https://manticoresearch.com)

precaching table 'events'
precaching table 'dev_events'

FATAL: out of memory (unable to allocate 18446744073709551615 bytes)

Indextool fails too:

# indextool --check dev_events
...
checking disk chunk, extension 0, 0(2)...
FATAL: out of memory (unable to allocate 18446744073709551615 bytes)

[sanikolaev]

The data to reproduce the issue can be found in /home/snikolaev/issue-20241201 on dev2.

[klirichek]

You would not start anyway after such upgrade, because columnar storage was upgraded, and from 5.0.2 it had internal v.9, and 6.3.8 has higher, and it should report something like

checking disk chunk, extension 0, 0(2)...
FAILED, disk chunk data/dev_events/dev_events.0: prealloc failed: Unable to load columnar storage: data/dev_events/dev_events.0.spc is v.9, binary is v.11
checking disk chunk, extension 1, 1(2)...
FAILED, disk chunk data/dev_events/dev_events.1: prealloc failed: Unable to load columnar storage: data/dev_events/dev_events.1.spc is v.9, binary is v.11

Unfortunately for your too ancient index behavior is broken. That is definitely should be fixed.
Anyway, it will be error message like cited; you can't upgrade.

[dzek69]

you can't upgrade.

wait, what? Can I upgrade gradually using some versions "in between"? Cause I kind of refuse to believe all of Manticore users at some point (maybe 5.0.2, maybe some later version) got locked with "you can't upgrade, start over" solution 😰

[klirichek]

@glookka m.b. it is ought to make kind of 'oneshot converter'?