Split segment by search type #2273
Conversation
| if (isShardLevelRescoringEnabled == true) { | ||
| ResultUtil.reduceToTopK(perLeafResults, firstPassK); | ||
| } | ||
|
|
||
| StopWatch stopWatch = new StopWatch().start(); | ||
| perLeafResults = doRescore(indexSearcher, leafReaderContexts, knnWeight, perLeafResults, finalK); | ||
| perLeafResults = doRescore(indexSearcher, knnWeight, perLeafResults, finalK); |
There was a problem hiding this comment.
Previously, we did exact search on finalK. After this change, we still does exact search on finalK. Could you tell me how will this improve the latency?
There was a problem hiding this comment.
doSearch can call either ApproxSearch or Exact Search based on conditions like whether engine files exists or not, number of docs after filter is less than k. In those cases, we will quantize query vector, and every vector from segments, and, then perform distance computation using Hamming distance for firstPassK. With this approach, we only call doSearch for those segments which we know will always call approxsearch, and, for other segments we will call exact search without quantization with finalK. The optimization is at https://github.com/opensearch-project/k-NN/pull/2273/files#diff-9cfe412357ba56b3ef216427d491fc653535686a760e8ba19ea1aa00fc0e0338R68-R78
There was a problem hiding this comment.
Are you assuming that an exact search on full precision vectors will be faster than an exact search with quantized vectors due to the slower quantization process? It would be interesting to see the benchmark results for this.
If that’s the case, an alternative could be to retrieve quantized values directly from the Faiss file instead of performing on-the-fly quantization.
There was a problem hiding this comment.
Yes, exact search on full precision for k is less than, exact search on quantization for first pass K + rescore matched docs on full precision . The linked GitHub issues actually shows how performance got impacted 10x when there are segments with no faiss engine files. In my POC, I saw improvements but recall was poor because of using order as link between results and leaf reader context. I am rerunning experiments with my change to collect metrics with latency and recall
There was a problem hiding this comment.
There is one case where we are running exact search; when the returned result is less than k. Are we going to handle that case as well?
There was a problem hiding this comment.
I believe this happens with filter, if so, yes, it was already taken care
There was a problem hiding this comment.
No. It happens regardless there is filter or not. https://github.com/opensearch-project/k-NN/blob/main/src/main/java/org/opensearch/knn/index/query/KNNWeight.java#L149
There was a problem hiding this comment.
No. It can happen either when there are no engine files or after filter that number of matched documents is less than k. We only decided to call doSearch if we know that it will call Approx Search API. For other segments we will directly call Exact Search, this PR is about that only https://github.com/opensearch-project/k-NN/pull/2273/files#diff-9cfe412357ba56b3ef216427d491fc653535686a760e8ba19ea1aa00fc0e0338R72-R75
VijayanB
force-pushed
the
split-segment-by-search-type
branch
2 times, most recently
from
235460f
to
229e788
Compare
We map order of results to order of segments, and finally rely on that order to build top docs. Refactor method to use map.Entry to map leafreader context with results from those leaves. This is required when we want to split segments based on approx search or exact search to reduce rescoring twice by exact search Signed-off-by: Vijayan Balasubramanian <balasvij@amazon.com>
For exact search, it is not required to perform qunatization during rescore with oversamples. However, to avoid normalization between segments from approx search and exact search, we will first identify segments that needs approxsearch and will perform oversamples and, at end, after rescore, we will add scores from segments that will perform exact search. Signed-off-by: Vijayan Balasubramanian <balasvij@amazon.com>
VijayanB
force-pushed
the
split-segment-by-search-type
branch
from
229e788
to
b149309
Compare
|
Should we skip quantization on indexing if we are not using it here then? |
+1. I think it is valid point. We should do that too. But @VijayanB there were couple of more ideas that we were discussing on how to fix this issue. Did you put some thoughts on that like why we should split the segments by search type? |
Description
For exact search, it is not required to perform
qunatization during rescore with oversamples.
However, to avoid normalization between segments from
approx search and exact search, we will first identify
segments that needs approxsearch and will perform oversamples
and, at end, after rescore, we will add scores from segments that
will perform exact search.
Related Issues
#2215
Check List
--signoff.By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.