这篇论文教你如何判断一个评测集靠不靠谱，用LLM当裁判自动挑毛病，省人工还靠谱。
该研究提出一种无参考评估框架，利用LLM评判员从一致性、复杂度和策略覆盖三个维度检查对话代理基准质量。框架与独立人工标注结果吻合，并在不同能力LLM生成的基准及受控降质扰动的基准上验证了区分效果。实验表明该指标能稳定区分基准质量等级，且适用于手工精选基准。
Benchmarking the Benchmarks: Evaluating Benchmarks for Conversational Agents
Task-oriented conversational agents are evaluated using curated or automatically generated benchmarks, yet benchmark quality is rarely assessed. Poor benchmarks may contain inconsistent tasks, simplistic scenarios, or limited policy coverage, leading to unreliable evaluations. We introduce a reference-free framework that uses LLM judges to assess benchmark consistency, complexity, and policy coverage, while providing actionable diagnostics of weaknesses. We validate the framework by demonstrating agreement with independent human annotations and by evaluating benchmarks generated by LLMs of varying capabilities, as well as benchmarks subjected to controlled quality-degrading perturbations. Across domains and judge models, the proposed metrics consistently distinguish between benchmark quality levels. We further demonstrate the framework's applicability to manually curated benchmarks. Our framework offers a practical approach for evaluating synthetic and manually curated conversational-agent benchmarks.