[feature request] Expose means of configuring/providing a custom Object Store extension

[alexkohler]

Hi there, I'm a user of lancedb, which leverages tantivy-py for full text search indices (see https://lancedb.github.io/lancedb/fts). A current shortcoming of the lancedb FTS support is that it's only supported for local file paths (something like S3 is not supported).

The lance maintainers have authored an Object Store extension , but as I understand it, there's no means of specifying/providing this extension within tantivy-py. Would love if this could be supported!

[changhiskhan]

LanceDB maintainer here. We'd be happy to contribute if you can point us to the right place to configure the underlying tantivy rust to integrate this extension. thanks!

[dgarnitz]

I'm also in need of this feature. Any idea when it will be schedule?

[dtiarks]

Also very much interested!

[erixison]

Also interested

[zmtbnv]

+1

[cjrh]

I'm looking at the example here: https://github.com/lancedb/tantivy-object-store/blob/main/examples/index_wiki_local.rs

    <snip>
    let dir = new_object_store_directory(
        Arc::new(LocalFileSystem::new()),
        dir.path().to_str().unwrap(),
        None,
        0,
        None,
        None,
    )
    .unwrap();

    let index_using_object_store =
        tantivy::Index::create(dir, schema.clone(), IndexSettings::default()).unwrap();
        
    let mut writer = index_using_object_store.writer(1024 * 1024 * 1024).unwrap();
    <snip>

The only difference is the dir object that is passed to Index::create(), is that correct?

[alex-au-922]

Hi LanceDB Developers, would you happy to create a PyO3 bindings in Python for your tantivy-object-store crate? I'm very willing to help

[cjrh]

@alex-au-922 are you currently working on this somewhere else? Or alternatively, @changhiskhan what is the current status on this from the lancedb pov?

[alex-au-922]

Nope, as I afraid there's some code plagarism issues so I haven't started. Also would like to see how LanceDB's view on this, as I think LanceDB is going to build their own retriever?