SUPERT: Unsupervised Multi-Document Summarization Evaluation & Generation

This project includes the source code for the paper SUPERT: Towards New Frontiers in Unsupervised Evaluation Metrics for Multi-Document Summarization, to appear at ACL 2020.

Highlighted Features

• Unsupervised evaluation metrics: Measure multi-document summaries without using human-written reference summaries
• Unsupervised multi-document summarizer: Using the unsupervised evaluation metrics as rewards to guide a neural reinforcement learning based summarizer to generate summaries.

Contact person: Yang Gao, yang.gao@rhul.ac.uk

https://sites.google.com/site/yanggaoalex/home

Don't hesitate to send us an e-mail or report an issue, if something is broken or if you have further questions

Example Use Cases

Evaluate Summaries (evaluate_summary.py)

Given the source documents and some to-be-evaluated summaries, you can produce the unsupervised metrics for the summaries with a few lines of code:

from ref_free_metrics.sbert_score_metrics import get_sbert_score_metrics
from utils.data_reader import CorpusReader

reader = CorpusReader('data/topic_1')
source_docs = reader()
summaries = reader.readSummaries() 
scores = get_sbert_score_metrics(source_docs, summaries, pseudo_ref='top15')

In the example above, it extracts the top-15 sentences from each source document to build the pseudo reference summaries, and rate the summaries by measuring their semantic similarity with the pseudo references.

Generate Summaries (generate_summary.py)

You can also use the unsupervised metrics as rewards to train a RL-based summarizer to generate summaries:

# read source documents
reader = CorpusReader()
source_docs = reader('data/topic_1')

# generate summaries, with summary max length 100 tokens
rl_summarizer = RLSummarizer()
summary = rl_summarizer.summarize(source_docs, summ_max_len=100)

# print out the generated summary
print(summary)

If human-written reference summaries are available (assume they are at data/topic_1/references), you can also evaluate the quality of the generated summary against the references using ROUGE:

refs = reader.readReferences() 
for ref in refs:
    rouge_scores = evaluate_summary_rouge(summary, ref)

How to Set Up

• Prerequisite: Python 3.6 or higher versions
• Install all packages in requirement.txt.

pip3 install -r requirements.txt

• (Optional, if you want to run ROUGE) Download ROUGE package from the link and place it in the rouge directory

mv RELEASE-1.5.5 rouge/

License

Apache License Version 2.0