EvalCards:为AI评估报告提供可解释层

Evaluation Cards: An Interpretive Layer for AI Evaluation Reporting

精选理由

AI评估报告混乱是行业痛点,做模型评测、写技术文档或选型决策的团队,可以直接用EvalCards统一报告格式,减少误解和重复劳动。

AI 摘要

AI评估结果虽大量产生,但报告格式不统一,导致读者难以跨来源比较、识别遗漏或追溯证据。现有方案仅覆盖评估生命周期的片段,缺乏统一记录,且未区分不同利益相关者的需求。研究者提出了EvalCards,一个可操作的报告层,整合基准元数据、评估运行数据和模型元数据。他们从52篇论文和10次访谈中推导出报告模式,实现了四个解释信号(可复现性、文档完整性、来源与风险、分数可比性),并针对研究与非研究受众设计了阅读模式。该工具已在5816个模型、635个基准和101843个结果上部署,揭示了当前报告实践中的系统性缺陷。

原文 · arXiv cs.AI

Evaluation Cards: An Interpretive Layer for AI Evaluation Reporting

AI evaluation results are produced at scale but reported inconsistently across leaderboards, model cards, benchmark papers, and company blogs. The cost is interpretive: readers cannot reliably compare results across sources, identify what a report omits, or trace an aggregate claim to its underlying evidence. Recent efforts address isolated components but leave three gaps: they cover only narrow slices of the evaluation lifecycle and do not compose into a single interpretable record; they specify static representations that do not differentiate the questions different stakeholders bring to the same evidence; and they remain proposals on paper, lacking the extraction infrastructure required for adoption at scale. We present \EvalCards{}, an operational reporting layer that composes benchmark metadata, evaluation run data, and model metadata into a unified record. We (1) derive a reporting schema from a structured review of 52 papers and 10 stakeholder interviews, (2) implement four interpretive signals (reproducibility, documentation completeness, provenance and risk, and score comparability), rendered through reader modes calibrated to research and non-research audiences, and (3) deploy a monitoring tool that applies \EvalCards{} across 5,816 models, 635 benchmarks, and 101,843 results, surfacing systematic gaps in current reporting practice.