这个基准直击LLM科学推理的软肋——从数据中归纳规律而非回忆知识，做AI评估或科学模拟的团队值得关注，它暴露了当前模型在长程推理和实验设计上的真实短板。
研究人员推出了DiscoverPhysics基准，通过让LLM代理在22个物理规则偏离现实的模拟世界中自主发现运动定律，来评估其科学推理能力。每个世界由N体模拟器按需生成，代理需设计多轮实验、观察原始轨迹数据，并提交自然语言解释和Python实现。测试发现，最强模型仅能通过一半世界，尤其在需要发现隐藏结构时失败；开源模型在实验设计和结论提取上显著落后于商业模型。该基准揭示了预测准确性与解释质量之间的差距，强调假设修正和实验设计对概念理解的重要性。
DiscoverPhysics: Benchmarking LLMs for Out-of-the-Box Scientific Thinking
Frontier LLMs now perform strongly across a wide range of physics evaluations, but it is hard to disentangle genuine reasoning from recall of established science. We introduce DiscoverPhysics, an interactive benchmark that asks a LLM agent to discover the laws of motion of a simulated world whose physics deliberately deviates from our own. We construct 22 worlds governed by, among others, screened and fractional-power gravity, multi-species couplings, hidden dark-matter-like particles, non-coordinate-free physics, and time-varying interactions. Each world is generated on demand by an N-body simulator, for which the agent proposes several rounds of experiments, observes raw trajectory data, and ultimately submits both a natural-language explanation of the world's physics and a Python implementation of the inferred law. Because solving a world requires the agent to design informative experiments and revise its hypotheses, the benchmark probes long-horizon reasoning over an experimental history. We evaluate submissions along two complementary axes: trajectory MSE on held-out particles and an LLM-judged explanation score following an expert-written rubric assessing conceptual understanding of each world. Across eleven frontier models, we find that the strongest agents pass only half of the worlds and consistently fail on those where latent structure must be uncovered. Open-source models lag substantially behind commercial models, both in their ability to design informative experiments and in extracting conclusions from the data. We further find that good predictive accuracy does not guarantee high explanation quality and that conceptual understanding depends on hypothesis refinement through well-chosen experiments.