多模型两两PK加Elo计分来评LLM输出，能调阈值控严格度，科学摘要上跟专家判断挺接近。
这篇论文提出用多个LLM对输出做两两比较、再用Elo评分生成排名的评估框架。通过可调的同意阈值，从全票一致到多数投票，控制评估置信度与覆盖率。作者在科学摘要生成的胜任力画像上做了验证，自动排名与专家判断相关性良好。相比人工评估显著减少干预成本，且不依赖参考标准。论文编号为arXiv:2607.28282v1。
(Towards) Scalable Reliable Automated Evaluation with Large Language Models
Evaluating the quality and relevance of textual outputs from Large Language Models (LLMs) remains challenging and resource-intensive. Existing automated metrics often fail to capture the complexity and variability inherent in LLM-generated outputs. Moreover, these metrics typically rely on explicit reference standards, limiting their use mostly to domains with objective benchmarks. This work introduces a novel evaluation framework designed to approximate expert-level assessments of LLM-generated content. The proposed method employs pairwise comparisons of outputs by multiple LLMs, reducing biases from individual models. An Elo rating system is used to generate stable and interpretable rankings. Adjustable agreement thresholds, from full unanimity to majority voting, allow flexible control over evaluation confidence and coverage. The method's effectiveness is demonstrated through evaluating competency profiles extracted from scientific abstracts. Preliminary results show that automatically derived rankings correlate well with expert judgments, significantly reducing the need for extensive human intervention. By offering a scalable, consistent, and domain-agnostic evaluation layer, the framework supports more efficient and reliable quality assessments of LLM outputs across diverse applications.