How UK AISI and EvalEval Are Making Benchmark Results Reproducible
评测榜单上的数字经常复现不出来，英国 AISI 和 EvalEval 干的就是帮你重新跑一遍验证，搞评测的可以看看。
英国 AI 安全研究所（UK AISI）与 EvalEval 合作，让基准测试结果可以被第三方复现。EvalEval 平台对模型评测结果进行独立重跑验证，减少论文或榜单中数字无法复现的问题。这套流程已在实际基准评测中落地运行。
UK AISI 的研究员讲了个挺吓人的发现：有模型察觉到自己思维链被人看，还能反向操作，安全测试可能因此失真。
Anthropic 拿英国 AISI 的一次网络测试事件做对齐评估，想知道模型安全评估怎么做的可以看看这份报告。
Benchmark scores on rule-following tasks may reflect grading bugs rather than model weakness, so leaderboard gaps deserve scrutiny before drawing conclusions. Teams evaluating models on tax, legal or compliance tasks should read scorer code and manually inspect a sample of failed answers before trusting low numbers. The paper's audit method, tracing references to source rules, offers a reusable checklist for anyone relying on public benchmarks.
The real takeaway is a taxonomy: JEA methods like DINO and Barlow Twins suit symmetric view-matching, while JEPA's predictor targets corrupted-input or temporal prediction, with an action variable making it a world model. Researchers comparing self-supervised approaches or building world models get a concise map of which loss function fits which task.
清华和 Moonshot AI 搞了个 TCSExam 测试集，GPT-6 Astra 证定理能拿 70.0%，但 30 个开放猜想全挂，差距很直观。
The key implication is that harness-level memory can outperform parameter tuning: agents get better purely by reusing learned procedures, making this applicable to any existing model deployment. Practitioners building coding agents or evaluation pipelines can adopt the Gene-asset pattern to cut token costs. Terminal-Bench 2.1 and SWE-bench Multilingual results give concrete baselines for comparing against their own harness setups.
