Integration with embedding library

[donhardman]

Proposal:

We should add parsing of the parameters to enable connection with the new library for automatic embeddings.

Currently, I suggest starting with the following specification:

• fields = "field1, field2": fields we should use as the source for text to generate embeddings
• model_name = "openai/..." or model_name = "sentence-transformers/..." for local models
• api_key = "...": for cases where we use a remote API for embeddings (we should be able to alter and change this one)
• use_gpu = 1|0: for cases where we use a local model and need to use GPU (if available), default is always CPU

Also we should consider to add new config:

• embedding_cache_path = ...: path to the cache directory where we will store everything in case of local model usage
  It may look like this:

CREATE TABLE test (
    title TEXT,
    image_vector FLOAT_VECTOR 
    KNN_TYPE='hnsw' 
    KNN_DIMS='4' 
    HNSW_SIMILARITY='l2'
    MODEL_NAME = "..."
);

Checklist:

^{To be completed by the assignee. Check off tasks that have been completed or are not applicable.}

Details

• Implementation completed
• Tests developed
• Documentation updated
• Documentation reviewed
• Changelog updated
• OpenAPI YAML updated and issue created to rebuild clients

[sanikolaev]

We may want to hide api_key in show create table output for security reasons.

[sanikolaev]

Dev call of Oct 28: ALTER should be supported for api_key

[glookka]

Adding new options is easy. However, it is not clear what should be done with these options

[sanikolaev]

@donhardman pls advise.

[donhardman]

We should parse it and store it somewhere to make it possible for the indexer stage to use it to generate embeddings with the lib as discussed before on call.

[donhardman]

Here are points to consider:

• We should create a model and store pointers linked to model_name. This means we need a map of pointers to use in the future when indexing, where the key in this map is model_name.
• Reference examples can be found in:
  • Test file: https://github.com/manticoresoftware/manticoresearch-text-embeddings/blob/main/examples/test.cpp
  • Header file: https://github.com/manticoresoftware/manticoresearch-text-embeddings/blob/main/manticoresearch_text_embeddings.h
• When model_name is specified, we should get dimensions from it by calling the get_hidden_size method and not allow users to pass their own knn_dims as it would create a conflict. Therefore, when model_name is specified, knn_dims is not required.
• The API key is validated during model creation by passing the model_name parameter. If the specified model requires an API key and one is not provided when creating a pointer, an error will be thrown.

The task primarily involves parsing rather than deep integration with the library.