Fix add_to_index overwrites and duplicate paths

[adharm]

This fixes the overwriting issue in add_to_index (from #71) and the issue with paths having duplicated sections (from #82).

I updated the tests to better cover the "add" functionality (moving them to their own file will come in a future PR).

I also added a few more items to the .gitignore.

[bclavie]

Thank you 🙏 I'll review in a bit.

Also, I thought you could be interested in checking out the other indexing approaches (flat & hnsw) and decoupling of model and index (see #158). This should make CRUD a lot easier to support, esp. for the non-PLAID indexes.

[bclavie]

LGTM except for one thing: It seems that when using the IndexUpdater, we never write the new collection to file, so there'll be errors when trying to load the index again and query (unless I'm missing something)

[jlscheerer]

Thank you very much @anirudhdharmarajan. I have integrated the changes in #158

[adharm]

First off, I'm super excited for the changes you've started off in #158, @jlscheerer! That's going to be incredibly useful, thanks for taking the lead on that.

@bclavie, good catch! I fixed that + consolidated repeated code and updated the path logic, lmk if these changes make sense.

[adharm]

Actually, just realized that the path logic needs updating, fixing now. Please hold!

[adharm]

I just reverted the logic for manipulating paths for the Searcher for now rather than changing anything for how paths are kept consistent (especially since index_root means different things here vs in the upstream colbert repo). I can tackle that in more detail once #158 goes through!
I'll also take a stab at restructuring the tests once that change goes through to avoid duplication of tests and expanding coverage.
Looks like huggingface is down again, so failed CI run 😞

[paulthemagno]

That would be super-useful!

[kevinningthu]

Thanks for your excellent work. May I ask when this PR will be merged into the main branch?

[akshay-anv]

Does this fix all the issues with add_to_index function ?

[paulthemagno]

Will the commit of this PR will be tagged as a new version available on pip during this days ?

[bclavie]

Hey @Akshay1921, not yet. These will be improved immensely by #158 which I'm expecting to land very soon.

@paulthemagno @kevinningthu Next version release will likely be Monday (potentially today, but might be a bit short on time) and will include this + #158, which will greatly improve a lot of aspects with indexing!

[kevinningthu]

Hey @Akshay1921, not yet. These will be improved immensely by #158 which I'm expecting to land very soon.

@paulthemagno @kevinningthu Next version release will likely be Monday (potentially today, but might be a bit short on time) and will include this + #158, which will greatly improve a lot of aspects with indexing!

Thank you guys so much!