Add download from remote feature #2301

ArthurChen189 · 2023-12-11T20:44:28Z

Sample command:

target/appassembler/bin/SearchCollection
-index msmarco-v1-passage
-topics tools/topics-and-qrels/topics.msmarco-passage.dev-subset.txt
-topicReader TsvInt
-output runs/run.msmarco-passage.bm25-default.topics.msmarco-passage.dev-subset.txt -bm25

src/main/java/io/anserini/index/IndexInfo.java

codecov · 2023-12-13T18:27:11Z

Codecov Report

Attention: 37 lines in your changes are missing coverage. Please review.

Comparison is base (2c14a49) 64.29% compared to head (6bbc664) 64.37%.

Files	Patch %	Lines
...in/java/io/anserini/util/PrebuiltIndexHandler.java	67.85%	14 Missing and 13 partials ⚠️
...main/java/io/anserini/search/SearchCollection.java	50.00%	6 Missing ⚠️
src/main/java/io/anserini/index/IndexInfo.java	89.18%	4 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master    #2301      +/-   ##
============================================
+ Coverage     64.29%   64.37%   +0.07%     
- Complexity     1333     1352      +19     
============================================
  Files           203      205       +2     
  Lines         11300    11432     +132     
  Branches       1426     1439      +13     
============================================
+ Hits           7265     7359      +94     
- Misses         3558     3583      +25     
- Partials        477      490      +13

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

src/main/java/io/anserini/util/IndexHandler.java

lintool · 2023-12-13T21:15:57Z

hi @ArthurChen189 can you give me sample commands to verify?

lintool · 2023-12-13T21:16:42Z

Linking to #2301

ArthurChen189 · 2023-12-13T21:18:17Z

Sorry, forgot to provide a sample command:

target/appassembler/bin/SearchCollection
-index msmarco-v1-passage
-topics tools/topics-and-qrels/topics.msmarco-passage.dev-subset.txt
-topicReader TsvInt
-output runs/run.msmarco-passage.bm25-default.topics.msmarco-passage.dev-subset.txt -bm25

lintool

round of comments.

src/main/java/io/anserini/index/IndexInfo.java

src/main/java/io/anserini/util/PrebuiltIndexHandler.java

lintool · 2023-12-13T23:45:01Z

WARNING?

[INFO] Running io.anserini.index.PrebuiltIndexHandlerTest
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.jline.terminal.impl.exec.ExecTerminalProvider (file:/Users/jimmylin/.m2/repository/org/jline/jline/3.23.0/jline-3.23.0.jar) to constructor java.lang.ProcessBuilder$RedirectPipeImpl()
WARNING: Please consider reporting this to the maintainers of org.jline.terminal.impl.exec.ExecTerminalProvider
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
msmarco-v1-passage 100% │██│ 2170758/2170758 (0:01:38 / 0:00:00) Downloading...
File downloaded successfully (MD5 check passed)!
msmarco-v1-passage 100% │██│ 2170758/2170758 (0:01:38 / 0:00:00) Downloading...
Decompressing index...
Index decompressed successfully!

Also, maybe something more lightweight for a unit test, like https://github.com/castorini/anserini-data/tree/master/CACM - lucene9-index.cacm.tar.gz?

This is what I use for some test cases that need a "real" index to download.

Equivalent test in Pyserini: https://github.com/castorini/pyserini/blob/master/tests/test_index_download.py

lintool · 2023-12-13T23:50:43Z

Very nice, works! 🎉

% target/appassembler/bin/SearchCollection \                                     
-index msmarco-v1-passage \
-topics tools/topics-and-qrels/topics.msmarco-passage.dev-subset.txt \
-topicReader TsvInt \
-output runs/run.msmarco-passage.bm25-default.topics.msmarco-passage.dev-subset.txt -bm25

msmarco-v1-passage 100% │█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████│ 2170758/2170758 (0:02:25 / 0:00:00) Downloading...
msmarco-v1-passage 100% │█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████│ 2170758/2170758 (0:02:25 / 0:00:00) Downloading...
Decompressing index...
Index decompressed successfully!
2023-12-13 18:49:02,174 INFO  [main] search.SearchCollection (SearchCollection.java:981) - ============ Initializing Searcher ============
2023-12-13 18:49:02,176 INFO  [main] search.SearchCollection (SearchCollection.java:982) - Index: /Users/jimmylin/.cache/pyserini/indexes/lucene-index.msmarco-v1-passage.20221004.252b5e
2023-12-13 18:49:02,302 INFO  [main] search.SearchCollection (SearchCollection.java:986) - Fields: []
2023-12-13 18:49:02,302 INFO  [main] search.SearchCollection (SearchCollection.java:697) - Using DefaultEnglishAnalyzer
2023-12-13 18:49:02,303 INFO  [main] search.SearchCollection (SearchCollection.java:698) - Stemmer: porter
2023-12-13 18:49:02,303 INFO  [main] search.SearchCollection (SearchCollection.java:699) - Keep stopwords? false
2023-12-13 18:49:02,303 INFO  [main] search.SearchCollection (SearchCollection.java:700) - Stopwords file: null
2023-12-13 18:49:02,342 INFO  [main] search.SearchCollection (SearchCollection.java:1258) - runtag: Anserini
2023-12-13 18:49:02,344 INFO  [main] search.SearchCollection (SearchCollection.java:1264) - ============ Launching Search Threads ============
2023-12-13 18:49:04,533 INFO  [pool-4-thread-5] search.SearchCollection$SearcherThread (SearchCollection.java:881) - ranker: bm25(k1=0.9,b=0.4), reranker: default: 100 queries processed
...

Although why does it show progress bar twice?

jasper-xian · 2023-12-15T03:12:04Z

Trying the command myself on orca, running into an error:

Index file already exists! Skip downloading.
Decompressing index...
Index decompressed successfully!
Index path 'msmarco-v1-passage' does not exist or is not a directory.

In my pyserini cache, I see the tarfile but no unzipped file, and the command runs pretty quickly, so seems to me that there was an issue in unzipping? I printed the paths to the tarfile and they seem to match up, as far as I'm aware.

Also, would it be possible to use an environment variable for PYSERINI_HOME similar to how python handles it? I'm not sure if this is possible in Java, but would be great as I have my cache set outside of my home directory on orca to save space.

ArthurChen189 · 2023-12-18T23:17:14Z

In my pyserini cache, I see the tarfile but no unzipped file, and the command runs pretty quickly, so seems to me that there was an issue in unzipping? I printed the paths to the tarfile and they seem to match up, as far as I'm aware.

Thanks for letting me know. I think that was due to an interruption during the download, therefore having an unchecked and incomplete tarball file. Could you delete it and run the commands again? Thanks!

ArthurChen189 · 2023-12-19T01:29:55Z

Got some unexpected errors after merging, changing this to draft

lintool · 2023-12-19T18:31:32Z

Tried the following and it worked!

target/appassembler/bin/SearchCollection \
  -index msmarco-v1-passage \
  -topics tools/topics-and-qrels/topics.msmarco-passage.dev-subset.txt \
  -topicReader TsvInt \
  -output runs/run.msmarco-passage.bm25-default.topics.msmarco-passage.dev-subset.txt -bm25

@ArthurChen189 let's go ahead and merge... we can circle back if there are additional issues.

In your commit msg, be sure to mention "pre-built indexes" somewhere...

jasper-xian · 2023-12-20T01:42:38Z

Tested the same command on my M1 Macbook Pro, worked for me as well!

target/appassembler/bin/SearchCollection \
  -index msmarco-v1-passage \
  -topics tools/topics-and-qrels/topics.msmarco-passage.dev-subset.txt \
  -topicReader TsvInt \
  -output runs/run.msmarco-passage.bm25-default.topics.msmarco-passage.dev-subset.txt -bm25

add download from remote feature

728e5fb

ArthurChen189 marked this pull request as draft December 11, 2023 20:44

lintool reviewed Dec 11, 2023

View reviewed changes

src/main/java/io/anserini/index/IndexInfo.java Show resolved Hide resolved

fix maven build

9c8267c

lintool reviewed Dec 13, 2023

View reviewed changes

src/main/java/io/anserini/util/IndexHandler.java Outdated Show resolved Hide resolved

ArthurChen189 added 3 commits December 13, 2023 19:53

add unittest for prebuilt index handler

2e12e02

rename IndexHandler to PrebuiltIndexHandler

f4f9de4

minor fixup

5962d92

ArthurChen189 marked this pull request as ready for review December 13, 2023 20:50

ArthurChen189 requested a review from lintool December 13, 2023 20:50

lintool reviewed Dec 13, 2023

View reviewed changes

src/main/java/io/anserini/index/IndexInfo.java Show resolved Hide resolved

src/main/java/io/anserini/index/IndexInfo.java Outdated Show resolved Hide resolved

src/main/java/io/anserini/util/PrebuiltIndexHandler.java Outdated Show resolved Hide resolved

This was referenced Dec 16, 2023

Add ability to download pre-built indexes #2296

Closed

Reproduce "End-to-End Retrieval with Learned Dense and Sparse Representations Using Lucene" with pre-built indexes #2307

Closed

minor fixup

288f95a

ArthurChen189 requested a review from lintool December 18, 2023 23:17

ArthurChen189 marked this pull request as draft December 19, 2023 01:30

merge with main

360f199

ArthurChen189 force-pushed the add-download branch from f539a25 to 360f199 Compare December 19, 2023 17:32

resolve merge conflicts

ded4ce6

ArthurChen189 marked this pull request as ready for review December 19, 2023 17:56

minor fixup

6bbc664

lintool approved these changes Dec 19, 2023

View reviewed changes

ArthurChen189 merged commit 45591ab into castorini:master Dec 19, 2023
3 checks passed

ArthurChen189 deleted the add-download branch December 19, 2023 18:53

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add download from remote feature #2301

Add download from remote feature #2301

ArthurChen189 commented Dec 11, 2023 •

edited

Loading

codecov bot commented Dec 13, 2023 •

edited

Loading

lintool commented Dec 13, 2023

lintool commented Dec 13, 2023

ArthurChen189 commented Dec 13, 2023

lintool left a comment

lintool commented Dec 13, 2023 •

edited

Loading

lintool commented Dec 13, 2023

jasper-xian commented Dec 15, 2023

ArthurChen189 commented Dec 18, 2023

ArthurChen189 commented Dec 19, 2023

lintool commented Dec 19, 2023

jasper-xian commented Dec 20, 2023

Add download from remote feature #2301

Add download from remote feature #2301

Conversation

ArthurChen189 commented Dec 11, 2023 • edited Loading

codecov bot commented Dec 13, 2023 • edited Loading

Codecov Report

lintool commented Dec 13, 2023

lintool commented Dec 13, 2023

ArthurChen189 commented Dec 13, 2023

lintool left a comment

Choose a reason for hiding this comment

lintool commented Dec 13, 2023 • edited Loading

lintool commented Dec 13, 2023

jasper-xian commented Dec 15, 2023

ArthurChen189 commented Dec 18, 2023

ArthurChen189 commented Dec 19, 2023

lintool commented Dec 19, 2023

jasper-xian commented Dec 20, 2023

ArthurChen189 commented Dec 11, 2023 •

edited

Loading

codecov bot commented Dec 13, 2023 •

edited

Loading

lintool commented Dec 13, 2023 •

edited

Loading