基准测试的排名可能因语义改写而失真，做模型评估或选型的团队值得关注——你的排名可能选错了赢家。
一项针对数学推理基准 MathCheck（ICLR 2025）的审计发现，其 129 个问题组中有 4 个（3.1%）语义错误的改写，移除后 GPT-4o 从第 2 名跌至第 4 名，Claude Haiku 和 DeepSeek V3 排名上升。跨模型一致性检查（至少 3/4 模型一致）能以不到 10 美元成本自动发现这些错误。在作者自己的数据集中，47% 的自动生成改写存在语义错误。更严重的是，Claude Haiku 4.5 准确率 86% 但语义一致性率（SCR）仅 50%，意味着半数定理在语义等价改写下答案不同；9 个前沿模型准确率差距仅 10 个百分点（86-96%），但 SCR 差距达 32 个百分点（50-82%）。论文提出 FormInv 协议，包括审计方法、SCR 和 Cochran's Q 度量，以及用于场景感知模型选择的 FormInvSelector。
FormInv: A Measurement Protocol for Semantic Invariance in Mathematical Reasoning Benchmarks
A paraphrase-quality audit of MathCheck (ICLR 2025) detected 4 semantically incorrect paraphrases in 129 groups (3.1%); removing them drops GPT-4o from rank 2 to rank 4 and elevates Claude Haiku and DeepSeek V3 above it; these ranking changes are invisible to any single-model evaluation. Cross-model unanimity found these errors automatically (>= 3/4 models for MathCheck; >= 6/9 for our primary evaluation) for under $10; in our own dataset the same protocol found that 47% of auto-generated connective-variation paraphrases were semantically incorrect. That flaw compounds a deeper measurement gap: Claude Haiku 4.5 achieves 86% accuracy yet SCR=50%, meaning half its theorems are answered differently under semantically equivalent restatements, while aggregate accuracy across 9 models spans only 86-96% yet Semantic Consistency Rates (SCR) span 50-82% -- a 32-point gap invisible to standard benchmarks. Formally, for any target ranking over 9 frontier models there exists a weighting over paraphrase families that realizes it (No-Free-Benchmark corollary), because no model Pareto-dominates all families -- so benchmark designers who select families are implicitly choosing which model wins. FormInv supplies the audit protocol (replicated on external benchmarks at 100% recall), SCR and per-theorem Cochran's Q as primary invariance measures evaluated on 9 models across 366-811 items (on Lean4-verified theorems), and FormInvSelector for regime-aware model selection.