yzha/ctc_eval

Metric Card for CTC_Eval

Metric Description

• —Previous work on NLG evaluation has typically focused on a single task and developed individual evaluation metrics based on specific intuitions.
• —In this work, we propose a unifying perspective based on the nature of information change in NLG tasks, including compression (e.g., summarization), transduction (e.g., text rewriting), and creation (e.g., dialog).
• —A common concept underlying the three broad categories is information alignment, which we define as the extent to which the information in one generation component is grounded in another.
• —We adopt contextualized language models to measure information alignment.

How to Use

Example:

    >>> ctc_score = evaluate.load("yzha/ctc_eval")
    >>> results = ctc_score.compute(references=['hello world'], predictions='hi world')
    >>> print(results)
    {'ctc_score': 0.5211202502250671}

Inputs

• —input_field
• —references: The document contains all the information
• —predictions: NLG model generated text

Output Values

The CTC Score.

Citation

@inproceedings{deng2021compression, title={Compression, Transduction, and Creation: A Unified Framework for Evaluating Natural Language Generation}, author={Deng, Mingkai and Tan, Bowen and Liu, Zhengzhong and Xing, Eric and Hu, Zhiting}, booktitle={Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing}, pages={7580--7605}, year={2021} }