统一了AI评估结果格式
论文提出Every Eval Ever，首个共享元数据模式和社区众包仓库，用于标准化AI评估结果。该模式将评估表示统一为单一JSON文档，支持从评价工具、论文等多种来源导入，并可存储每个实例的输出以进行细粒度分析。当前社区数据库已包含22,235个模型、2,273个独特基准和31种评估格式。论文还提供了自动转换器，从流行格式和评价工具转换到统一模式。
Every Eval Ever: A Unifying Schema and Community Repository for AI Evaluation Results
AI evaluations are widely used for testing and understanding progress. However, the diverse evaluators bring with them inconsistencies that challenge analysis and comparison. First, results are saved in incompatible formats, scattered across leaderboards, papers, blog posts, evaluation harness logs, and custom repositories. Second, results are created by different evaluation frameworks, which produce divergent scores for nominally identical evaluations and record metadata inconsistently, hindering comparison, cross-community evaluation science, cost reduction, and reuse. We introduce Every Eval Ever, the first shared schema and community-crowdsourced repository for AI evaluation results. The schema standardizes how evaluations are represented in a unified, single JSON document. It is source-agnostic by design, ingesting results from evaluation harnesses and papers alike, and optionally stores per-instance outputs for fine-grained analysis. We contribute: (i) a community-governed metadata schema with a companion instance-level schema, the first standardization effort of its kind; (ii) automatic converters from popular formats, evaluation harnesses, and leaderboards to the unified schema; and (iii) a crowdsourced community database hosted on Hugging Face, currently spanning to date 22,235 models, 2,273 unique benchmarks, and 31 evaluation formats.