AISI用更高的token预算测试模型，发现性能远超标准评测——8%的任务需要千万级token才能解决，前沿模型能力被低估了。
AISI在测试前沿模型时，将推理计算预算从250万token提高到5000万token，模型的时间跨度从40分钟跃升至4小时。在网络安全、软件工程和数学基准上，模型性能随token预算增加而持续提升，8%的任务仅在1000万token以上才被解决。新模型更能有效利用额外计算资源，前沿模型的加倍率在高预算下陡增60%。标准评估因在模型性能还未饱和前就限制计算量，系统性低估了模型能力，固定预算的得分会遗漏大量提升。
One frontier model's time horizon jumped from 40 m…
One frontier model's time horizon jumped from 40 minutes to 4 hours when AISI raised its compute budget from 2.5M to 50M tokens.
AISI tested frontier models across cyber, software engineering and maths benchmarks with higher token budgets than standard evals allow. Performance kept rising - 8% of tasks were only solved at 10M+ tokens - and newer models made better use of the extra compute. The frontier doubling rate measured 60% steeper at higher budgets.
Standard evals systematically underestimate capability by capping compute before models plateau. The longest and hardest tasks get cut off first, so fixed-budget scores miss substantial gains. https://t.co/Cgl4pSg4B9
- Decoder07-03 16:14原文