这篇论文搞了个叫Maskability Index的指标,帮你省掉试不同提示的功夫,直接判断填空提示还是前缀提示更管用,在ATOMIC2020上验证过了。
论文提出Maskability Index (MI)度量,基于DepthRank分数差异计算,用于判断在T5和BERT等预训练语言模型上,针对知识关系生成,masked-style提示是否优于prefix-style提示。在ATOMIC2020知识库补全基准上评估,MI与下游生成性能呈正相关。该指标在低资源场景下可辅助提示模板选择与适配策略。
The Maskability Index: Predicting Task-Objective Alignment in Pretrained Language Models
Large-scale pretrained language models such as T5 and BERT have demonstrated strong capabilities for generating structured knowledge. However, their performance depends on how closely the prompting strategy matches the objectives used during pretraining. We introduce the Maskability Index (MI), a quantitative metric that estimates whether a knowledge relation is better suited to masked-style prompting or prefix-style prompting in few-shot generation. MI is computed from differences in DepthRank scores between masked and unmasked templates, providing a principled measure of objective-template alignment. We evaluate MI on a diverse set of relations from the ATOMIC2020 knowledge base completion benchmark and show that it is positively correlated with downstream generation performance. These results indicate that MI can help select appropriate prompting templates and adaptation strategies for extracting relational knowledge from pretrained language models, especially in low-resource settings.