PACE-Bench:动态环境下基于代码演进的物理适应基准

PACE-Bench: Benchmarking Physics Adaptation via Code Evolution in Dynamic Environments

精选理由

想看看AI在规则变了之后能不能自己改代码?PACE-Bench 专门测这个,目前最强的 GPT-5.5 也只解决了 Statics 子集的三分之二,离满分还远得很。

AI 摘要

PACE-Bench 是一个基于模拟器的基准测试,包含144个源到目标的物理适应对,覆盖6个物理域。每个任务要求代理在环境突变后,通过代码演进将源码改造成目标代码。在10种自进化方法的对比中,Reflexion + Qwen3-14B 在完整基准上仅成功35.9%,而 GPT-5.5 在 Statics 子集上达到66.7%的成功率。结果显示,基于模拟器的反馈修正比未经验证的自修订更可靠,但整体性能尚未饱和。

原文 · arXiv cs.AI

PACE-Bench: Benchmarking Physics Adaptation via Code Evolution in Dynamic Environments

Self-evolving agents improve future behavior from interaction experience, yet existing evaluations typically optimize under fixed execution conditions and do not test recovery after those conditions change. To address this gap, we introduce PACE-Bench (Physics Adaptation via Code Evolution), a simulator-grounded benchmark of 144 source-to-target adaptation pairs across six physics domains. Each pair links a source environment to a mutated target environment with the same goal and interface. A code-driven design that succeeds in the source fails in the target, where agents must iteratively adapt it into a working target design using diagnostic sandbox feedback within a limited attempt budget. We compare ten self-evolving methods from four paradigms. The benchmark remains far from saturated: Reflexion + Qwen3-14B succeeds on only 35.9\% of full-benchmark pairs, while GPT-5.5 solves 66.7\% of the Statics subset under the full budget. Together, these results show that simulator-grounded reflection is more reliable than unverified self-revision, while memory anchors agents to early designs and broad tree search explores without converging. Even revealing exact physical changes does not raise the performance ceiling, pointing to mechanism redesign rather than parameter inference as the central bottleneck. Data and code are available at https://github.com/thunlp/PACE-Bench.