BridgeAI-Lab/Sem-nCG

1---2title: Sem-nCG3tags:4- evaluate5- metric6description: "Sem-nCG (Semantic Normalized Cumulative Gain) Metric evaluates the quality of predicted sentences 7(abstractive/extractive) in relation to reference sentences and documents using Semantic Normalized Cumulative Gain 8(NCG). It computes gain values and NCG scores based on cosine similarity between sentence embeddings, leveraging a 9Sentence-BERT encoder. This metric is designed to assess the relevance and ranking of predicted sentences, making it 10useful for tasks such as summarization and information retrieval."11sdk: gradio12sdk_version: 3.19.113app_file: app.py14pinned: false15authors:16- user: nbansal17---18 19# Metric Card for Sem-nCG20 21## Metric Description22Sem-nCG (Semantic Normalized Cumulative Gain) metric evaluates system-generated summaries (`predictions`) by comparing 23them with ground truth reference summaries (`references`) and input documents (`documents`). It computes the Semantic 24Normalized Cumulative Gain (NCG) scores based on sentence embeddings, which assess the quality of summaries by 25evaluating the relevance of predicted sentences to the reference and input document sentences.26 27 28## How to Use29 30Before using this metric, you need to install the dependencies:31```bash32pip install -U evaluate sentence-transformers nltk33```34 35#### Python Usage36```python37from evaluate import load38predictions = [39    "This is a prediction1 sentence 1. This is a prediction1 sentence 2.", 40    "This is a prediction2 sentence 1."41]42references = [43    "This is a reference1 sentence 1. This is a reference1 sentence 2.",44    "This is a reference2 sentence 1. This is a reference2 sentence 2."45]46documents = [47    "This is a document1 sentence 1. This is a document1 sentence 2. This is a document1 sentence 3.",48    "This is a document2 sentence 1. This is a document2 sentence 2."49]50model_name = "all-MiniLM-L6-v2"51metric = load("nbansal/semncg", model_name=model_name)  # model_name is optional. Default=all-MiniLM-L6-v252mean_score, scores = metric.compute(predictions=predictions, references=references, documents=documents)53print(f"Mean SemnCG: {mean_score}")54```55 56First step is to initialize the metric as `metric = load("nbansal/semncg", model_name=model_name)` where `model_name` is 57the sentence embedding model. The default value is `all-MiniLM-L6-v2`.58 59To `compute` the Sem-nCG scores, you need to provide three mandatory arguments:60 - `predictions` - List of predictions61 - `references` - List of references62 - `documents` - List of input documents63 64Sem-nCG also accepts several optional arguments:65 - `tokenize_sentences (bool)`: Flag to indicate whether to tokenize the sentences in the input documents. Default: True66 - `pre_compute_embeddings (bool)`: Flag to indicate whether to pre-compute embeddings for all sentences. Default=False67 - `k (int)`: The rank threshold used for evaluating gains (typically top-k sentences). Default is 3.68 - `gpu (Union[bool, str, int, List[Union[str, int]]])`: Whether to use GPU, CPU, or multiple processes for computation.69 - `batch_size (int)`: Batch size for encoding. Default is 32.70 - `verbose (bool)`: Flag to indicate verbose output. Default is False.71 - `debug (bool)`: Flag to return detailed debug information including ranked gains. Default is False.72 73Refer to the inputs descriptions for more detailed usage as follows:74```python75import evaluate76metric = evaluate.load("nbansal/semncg")77print(metric.inputs_description)78```79 80### Output Values81 82The output is a tuple containing:83 84- Mean Sem-nCG score: float: The average Sem-nCG score. 85- scores: List[Union[float, RankedGains]]: List of Sem-nCG scores or RankedGains objects for each document.86 87 88## Extensions89The current implementation supports any model from Huggingface/SentenceTransformer that is compatible with 90SentenceTransformer, such as `all-mpnet-base-v2` or `roberta-base`. You can extend the metric with more models by 91extending the `Encoder` base class in the `encoder_models.py` file.92 93## Deviations from Published Methodology94In our implementation, we expand upon the methodology presented in the original paper, which focused solely on 95extractive model summaries. The primary approach in the paper involved ranking sentences in the source document based on96ground-truth reference sentences. The Normalized Cumulative Gain (NCG) score was computed using the formula:97 98```ncg = $\frac{\text{cumulative gain}}{\text{ideal cumulative gain}}$```99 100Key deviations in our implementation from the paper include:1011. **Inclusion of Abstractive Model Summaries:** Unlike the paper, which exclusively considered extractive model 102summaries, our implementation supports both extractive and abstractive summarization models. 1032. **Enhanced Calculation of NCG Scores:** For both extractive and abstractive summaries, we compute rankings based on 104both the reference/ground truth (`gt_gain`) and predicted summaries (`pred_gain`). The NCG score is calculated using the105method shown below:106```python107def compute_ncg(pred_gains, gt_gains, k: int) -> float:108    gt_dict = dict(gt_gains)109    gt_rel = [v for _, v in gt_gains[:k]]110    model_rel = [gt_dict[position] for position, _ in pred_gains[:k]]111    return sum(model_rel)/sum(gt_rel)112```113 114This approach allows us to evaluate summarization quality across both extractive and abstractive methods, providing a 115more comprehensive assessment than the original methodology.116 117 118## Citation119```bibtex120@inproceedings{akter-etal-2022-revisiting,121    title = "Revisiting Automatic Evaluation of Extractive Summarization Task: Can We Do Better than {ROUGE}?",122    author = "Akter, Mousumi  and123      Bansal, Naman  and124      Karmaker, Shubhra Kanti",125    editor = "Muresan, Smaranda  and126      Nakov, Preslav  and127      Villavicencio, Aline",128    booktitle = "Findings of the Association for Computational Linguistics: ACL 2022",129    month = may,130    year = "2022",131    address = "Dublin, Ireland",132    publisher = "Association for Computational Linguistics",133    url = "https://aclanthology.org/2022.findings-acl.122",134    doi = "10.18653/v1/2022.findings-acl.122",135    pages = "1547--1560",136    abstract = "It has been the norm for a long time to evaluate automated summarization tasks using the popular ROUGE metric. Although several studies in the past have highlighted the limitations of ROUGE, researchers have struggled to reach a consensus on a better alternative until today. One major limitation of the traditional ROUGE metric is the lack of semantic understanding (relies on direct overlap of n-grams). In this paper, we exclusively focus on the extractive summarization task and propose a semantic-aware nCG (normalized cumulative gain)-based evaluation metric (called Sem-nCG) for evaluating this task. One fundamental contribution of the paper is that it demonstrates how we can generate more reliable semantic-aware ground truths for evaluating extractive summarization tasks without any additional human intervention. To the best of our knowledge, this work is the first of its kind. We have conducted extensive experiments with this new metric using the widely used CNN/DailyMail dataset. Experimental results show that the new Sem-nCG metric is indeed semantic-aware, shows higher correlation with human judgement (more reliable) and yields a large number of disagreements with the original ROUGE metric (suggesting that ROUGE often leads to inaccurate conclusions also verified by humans).",137}138```139 140## Further References141 - [Paper](https://aclanthology.org/2022.findings-acl.122/)142 - [Video](https://underline.io/lecture/50182-findings-revisiting-automatic-evaluation-of-extractive-summarization-task-can-we-do-better-than-rougequestion)143