论文提出了三个新指标，能清楚区分提取式和抽象式摘要，还能帮你发现幻觉问题，比ROUGE更深入。
该研究提出了三个抽象性度量指标：Reference Abstraction (RA)、Summary Abstraction (SA) 和 Abstraction Ratio (AR)，用调和均值与立方非重叠因子量化摘要偏离原文摘抄的程度。在100篇XSUM文档上对BART-large-cnn、Pegasus-xsum、DistilBart、MT5-small四个模型测试，SA在提取式模型（0.12-0.26）与抽象式模型（0.96-1.77）间有显著区分。AR指标还能识别需人工核查幻觉的摘要。代码和数据开源在GitHub。
Abstractiveness Metrics for Evaluating Text Summarization: A Refined Formulation with Empirical Validation
Quantifying abstractiveness in generated summaries is essential for evaluating summarization models beyond surface-level metrics like ROUGE. We introduce Reference Abstraction (RA), Summary Abstraction (SA), and Abstraction Ratio (AR) -- a set of principled heuristic metrics that measure how much a summary diverges from extractive copying of the source text. The formulation uses the harmonic mean of document lengths modulated by a cubic non-overlap factor, yielding dimensionally consistent, bounded output with non-linear sensitivity to the extractive-abstractive boundary. Evaluation on 100 XSUM documents across four summarization models (BART-large-cnn, Pegasus-xsum, DistilBart, MT5-small) demonstrates that the metrics successfully discriminate between extractive models (SA ~ 0.12-0.26) and abstractive models (SA ~ 0.96-1.77), and that the Abstraction Ratio identifies summaries requiring manual evaluation for potential hallucination. Code and results are available at https://github.com/katweNLP/AbstractionStudy.