Skip to content

๐Ÿ“ Reference-Free automatic summarization evaluation with potential hallucination detection

Notifications You must be signed in to change notification settings

Muhtasham/summarization-eval

Folders and files

NameName
Last commit message
Last commit date

Latest commit

ย 

History

23 Commits
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 

Repository files navigation

Summarization Evaluation

Elegantly automate text summarization evaluation in reference-free manner with potential hallucination detection.

Demo 1 Demo 2

Features

  • Easy to Use: Simply provide a text file containing the summary to be evaluated, and the script will handle the rest.
  • Custom Metrics: Utilize metrics such as word overlap and Jaccard similarity for in-depth content analysis.
  • Hallucination Detection: Systematically identify hallucinated content in summaries by analyzing semantic discrepancies with the original text, also use new word detection to identify potentially hallucinated summaries.If you came here for this just check detect_hallucinations under src/utils.py.
  • GPT-based Evaluation: Use GPT for nuanced qualitative assessments of summaries which takes care of json enforcement with pydantic, for easier parsing.
  • Adapted ROUGE & BERTScore: Rework traditional metrics for use in a reference-free context, focusing on the intrinsic qualities of summaries, as highlighted in Eugene Yan's writing.
  • Extensible: Easily add new metrics and models to the project to expand its capabilities, open an issue or a PR if you want to add something.

Installation

Clone the repository and install dependencies:

git clone https://github.com/Muhtasham/summarization-eval
cd summarization-eval
pip install -r requirements.txt

Usage

To use the main.py script under src, simply provide a text file containing the text you want to summarize. The script reads the file, generates summary, processes the summary, and outputs the evaluation results in a structured and readable format in the terminal.

Example:

python main.py --file-path "assets/news.txt"

Note: You will need to have an OpenAI API key set up in your environment to run the script.

Contributing

Contributions to enhance and expand this project are welcome. Please see the CONTRIBUTING.md file for guidelines on how to contribute.

License

This project is licensed under the MIT License.

About

๐Ÿ“ Reference-Free automatic summarization evaluation with potential hallucination detection

Topics

Resources

Stars

Watchers

Forks

Languages