The best way to evaluate your embedding models!
IMPORTANT: rankwise is in an early stage of development; the interface may change without further notice.
rankwise is a benchmark CLI tool for evaluating the quality of embedding models with your own dataset.
It also facilitates the generation of datasets using LLM models.
- Install dependencies:
poetry install- Activate the Poetry Shell:
poetry shell- Run the
generatecommand to create a dataset containing queries and their related documents in JSONL format from adata.jsonlfile with one document (a json encoded string) per line.
API_KEY=xxx rankwise generate --model "azure_openai.AzureOpenAI(model='gpt-4o',deployment_name='gpt-4o',api_version='2023-07-01-preview',azure_endpoint='https://your-azure-endpoint',api_key=ENVVAR('API_KEY'))" --queries-count 3 --input data.jsonl > dataset.jsonlThis command uses the given LLM model to generate the specified number of queries for every document in the input file.
- Run the
evaluatecommand to assess your dataset and obtain quality metrics for the specified embedding model.
API_KEY=xxx rankwise evaluate -E "azure_openai.AzureOpenAIEmbedding(model='text-embedding-3-large',deployment_name='azure-text-embedding-ada-002',api_version='2023-07-01-preview',azure_endpoint='https://your-azure-endpoint',api_key=ENVVAR('API_KEY'))" -m hit_rate -m mrr --input dataset.jsonlThis command uses the given embedding model to evaluate the input dataset and calculate quality metrics.
Contributions are what make the open-source community such an amazing place to learn, inspire, and create. Any contributions you make are greatly appreciated.
Please read the CONTRIBUTING file for details on our code of conduct and the process for submitting pull requests.
- Roberto Abdelkader Martínez Pérez
- Pedro Ruiz Pareja
- 0.1.0
- Initial Release
This project is licensed under the Apache v2.0 License - see the LICENSE file for details
- Thanks to the open-source community for continuous inspiration.
- Special mentions to contributors and collaborators.
If you have any questions or suggestions, feel free to contact the authors.