Papers

实证审计:Cosmos Policy 上测试时搜索何时有效

When Does Test-Time Search Help World Action Models? An Empirical Audit of Candidate Selection and Simulator Integrity

Why we picked it

这篇论文给测试时搜索泼了盆冷水:在 Cosmos Policy 上,价值选择器居然比随机还差,剪枝后端到端也只快 1.15 倍,还揪出个渲染 bug,做机器人策略的都该看看。

这项研究以 Cosmos Policy 和 RoboCasa 为对象,用预注册方法审计测试时缩放的三个前提条件。结果显示学习到的价值选择器比随机选择还差 1.66 个百分点,事后 oracle 也只多 1.40 个百分点,82% 的状态中候选动作结果完全相同,几乎没有选择空间。Token 剪枝能将单次前向 FLOPs 降低最多 47%,但端到端实测加速只有 1.15 倍,低于 1.5 倍的目标。研究还发现并修复了一个渲染损坏问题,其发生率沿计算密集的规划路径急剧上升。

Original · arXiv cs.AI

When Does Test-Time Search Help World Action Models? An Empirical Audit of Candidate Selection and Simulator Integrity

World-action models increasingly apply test-time scaling, spending extra inference computation on candidate selection and adaptive allocation to improve control. Such gains are credible only when candidates carry exploitable signal, nominal compute savings become real speedup, and evaluation stays reliable. We audit all three on Cosmos Policy and RoboCasa with a premise-first, pre-registered methodology. On clean data, a learned value selector is 1.66 pp worse than random; a hindsight oracle gains only 1.40 pp, with an episode-level interval spanning zero; and 82% of states show identical candidate outcomes, leaving little selection headroom. Token pruning cuts per-forward FLOPs by up to 47% but yields at most a measured 1.15$\times$ end-to-end speedup on the deployed path (1.25$\times$ on a decode-excluded path), short of a 1.5$\times$ target, and our adaptive-allocation experiment is inconclusive under a power gate. We further identify and fix a silent rendering corruption whose rate rises sharply along compute-heavy planning paths while leaving the official direct-policy result unaffected, and release a detector, fix, and scripts. These findings show why test-time scaling should be audited as a chain of premises (decision signal, realized efficiency, evaluationintegrity) rather than by success rate alone. Conclusions are scoped to the tested configuration.