-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
1 parent
58e579b
commit c6c323a
Showing
3 changed files
with
61 additions
and
1 deletion.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,42 @@ | ||
from flashrank import Ranker, RerankRequest | ||
|
||
# Nano (~4MB), blazing fast model & competitive p | ||
ranker = Ranker() | ||
# query = "How to speedup LLMs?" | ||
# passages = [ | ||
# { | ||
# "id":1, | ||
# "text":"Introduce *lookahead decoding*: - a parallel decoding algo to accelerate LLM inference - w/o the need for a draft model or a data store - linearly decreases # decoding steps relative to log(FLOPs) used per decoding step.", | ||
# "meta": {"additional": "info1"} | ||
# }, | ||
# { | ||
# "id":2, | ||
# "text":"LLM inference efficiency will be one of the most crucial topics for both industry and academia, simply because the more efficient you are, the more $$$ you will save. vllm project is a must-read for this direction, and now they have just released the paper", | ||
# "meta": {"additional": "info2"} | ||
# }, | ||
# { | ||
# "id":3, | ||
# "text":"There are many ways to increase LLM inference throughput (tokens/second) and decrease memory footprint, sometimes at the same time. Here are a few methods I’ve found effective when working with Llama 2. These methods are all well-integrated with Hugging Face. This list is far from exhaustive; some of these techniques can be used in combination with each other and there are plenty of others to try. - Bettertransformer (Optimum Library): Simply call `model.to_bettertransformer()` on your Hugging Face model for a modest improvement in tokens per second. - Fp4 Mixed-Precision (Bitsandbytes): Requires minimal configuration and dramatically reduces the model's memory footprint. - AutoGPTQ: Time-consuming but leads to a much smaller model and faster inference. The quantization is a one-time cost that pays off in the long run.", | ||
# "meta": {"additional": "info3"} | ||
|
||
# }, | ||
# { | ||
# "id":4, | ||
# "text":"Ever want to make your LLM inference go brrrrr but got stuck at implementing speculative decoding and finding the suitable draft model? No more pain! Thrilled to unveil Medusa, a simple framework that removes the annoying draft model while getting 2x speedup.", | ||
# "meta": {"additional": "info4"} | ||
# }, | ||
# { | ||
# "id":5, | ||
# "text":"vLLM is a fast and easy-to-use library for LLM inference and serving. vLLM is fast with: State-of-the-art serving throughput Efficient management of attention key and value memory with PagedAttention Continuous batching of incoming requests Optimized CUDA kernels", | ||
# "meta": {"additional": "info5"} | ||
# } | ||
# ] | ||
|
||
def rank_query_passages(ranker, query, passages): | ||
rerankrequest = RerankRequest(query=query, passages=passages) | ||
results = ranker.rerank(rerankrequest) | ||
print(results) | ||
return results | ||
|
||
def rank_query_lambda_handler(event, context): | ||
return rank_query_passages(ranker, event['query'], event['passages']) |