谁给评分者评分？面向自提升LLM智能体共进化评估指标与技能
Who Grades the Grader? Co-Evolving Evaluation Metrics and Skills for Self-Improving LLM Agents
这篇论文解决了自进化AI的一个核心漏洞：没有可靠指标怎么办。他们让指标和技能一起进化，在多个基准上验证了有效性，还能防作弊。值得研究Agent自改进的人读一读。
这篇论文指出，自进化智能体系统依赖一个隐藏假设：存在可靠的评估指标，但现实中往往没有。作者提出演化指标方法，通过搜索小缺陷检测器的组合形成可检查的指标，使用十项锚定参考集训练，并采用共识正则化和留出审计。
在代码生成（MBPP+）、企业文本到SQL（Spider 2.0-Snow）和无参考报告生成任务中，共进化方法Double Ratchet保留了由真实指标驱动下的88-110%的提升效果。
安全方面，移除锚定守卫会导致指标退化，独立裁判能捕获技能游戏行为，且任务感知裁判在77%的判定对中偏好演化输出。
Who Grades the Grader? Co-Evolving Evaluation Metrics and Skills for Self-Improving LLM Agents
Self-evolving agent systems improve by creating, revising, and retiring their own skills, but every such loop rests on a hidden assumption: a reliable evaluation metric already exists. In many real applications it does not. We make three claims. First, metrics can be \emph{evolved}: our metric loop searches compositions of small drawback detectors under a full evolutionary lifecycle, trained to agree with a ten-item anchored reference set, regularized by consensus over unlabeled outputs, and audited against a held-out anchor it never reads, yielding a transparent, inspectable metric rather than an opaque judge. Second, since no metric exists to beat, the yardstick is recovering what an accurate metric would have enabled, and \emph{Double Ratchet}, our co-evolution of the metric with a lifecycle-managed skill loop, does so: across code generation (MBPP+), enterprise text-to-SQL (Spider~2.0-Snow), and reference-free report generation, it retains 88--110\% of the held-out lift achieved by the same skill loop driven by ground truth or the best available rubric. Third, safety comes from anchor discipline plus outer audits: removing anchor guards collapses the metric into a vacuous detector while removing the lifecycle does not; and when evolved skills gamed the report rubric, an independent judge caught it, one detector repaired it, and a task-aware judge then preferred the evolved outputs over the pre-evolution baseline in 77\% of decided pairs. We argue this failure-expecting architecture is the right default wherever no reliable automatic verifier exists.
This report is categorized under AITOP's "Papers" desk. Key entities and technical topics under continuous tracking include LLM, 智能体, 评估指标, 共进化.
Source attribution: curated from arXiv cs.AI (@Xing Zhang, Guanghui Wang, Yanwei Cui, Ziyuan Li, Wei Qiu, Bing Zhu, Peiyang He).