Improve errormessage Failed to download model weights

[cappadaan]

Proposal:

Currently I get in Manticore 17.5

prealloc: Failed to download model weights

while trying to use a bigger model (eg Qwen 4B).

I have no idea why I get this message, it doesnt explain the problem. The available RAM is 80G+, so I should be able to run most models.

Request: improve this errormessage:

• why can I not download the weight
• what is required for resources
• what I currently have for resources

Checklist:

^{To be completed by the assignee. Check off tasks that have been completed or are not applicable.}

Details

• Implementation completed
• Tests developed
• Documentation updated
• Documentation reviewed
• OpenAPI YAML updated and issue created to rebuild clients

[sanikolaev]

@cappadaan please check if this works for you https://github.com/manticoresoftware/manticoresearch/blob/ef4d1e6cf1017c9db3be91d8e23d9644e5c9ad3e/test/clt-tests/mcl/auto-embeddings-qwen.rec#L22

[cappadaan]

@sanikolaev Yes, 0.6 works. But 4B/6B do not work and gives this error.

[sanikolaev]

Related PR #131

[sanikolaev]

Support for Qwen3-Embedding-4B (and other models that use sharded safetensors) was added in MCL version 10.2.2:
https://github.com/manticoresoftware/columnar/releases/tag/release-10.2.2

It now works as expected.

mysql -P9306 -h0 -v -e "drop table if exists test; CREATE TABLE test (f text, v FLOAT_VECTOR KNN_TYPE='hnsw' HNSW_SIMILARITY='l2' MODEL_NAME='Qwen/Qwen3-Embedding-4B' FROM='f'); insert into test(f) values('bread'),('cat'); select f, knn_dist() from test where knn(v, 'kitten')"
--------------
drop table if exists test
--------------

--------------
CREATE TABLE test (f text, v FLOAT_VECTOR KNN_TYPE='hnsw' HNSW_SIMILARITY='l2' MODEL_NAME='Qwen/Qwen3-Embedding-4B' FROM='f')
--------------

--------------
insert into test(f) values('bread'),('cat')
--------------

--------------
select f, knn_dist() from test where knn(v, 'kitten')
--------------

+-------+------------+
| f     | knn_dist() |
+-------+------------+
| cat   | 0.22313836 |
| bread | 0.56796616 |
+-------+------------+

You need to update the columnar-lib package, e.g.:

sudo apt install manticore-columnar-lib

I’m keeping this issue open so I can improve the error message in other cases.

[cappadaan]

thx @sanikolaev I can confirm it works now.