CAE-Bench 发布:评估 AI 智能体开发可复用 CAE 求解器的能力
CAE-Bench: Benchmarking AI Agents for Reusable CAE Solver Development
搞 CAE 仿真的工程团队和做智能体评测的研究者应该看看。它专挑求解器泛化能力下手,80 个任务、12 个模型测下来,连 GPT-6 Astra 在流体任务上都翻车,比通用代码基准更能暴露智能体的真实短板。附带了数据格式和 SDK,可以直接接进自己的测试环境。
研究者发布 CAE-Bench,一个用于评估 AI 智能体开发计算机辅助工程(CAE)求解器能力的基准。CAE 求解器不只是 PDE 求解器,还需要在不同边界条件、几何形状和材料属性等输入上保持泛化。基准的设计方式是:同一任务(如传热问题)提供多个用例,智能体只能使用少量公开用例开发求解器,然后在从未见过的保留用例上测试。
实验覆盖 12 个模型、80 个计算任务和多种推理设置。结果显示,AI 开发的求解器即使公开用例上表现良好,在保留用例上也可能得 0 分;在流体动力学任务上,连 GPT-6 Astra 开发的求解器在保留用例上也表现很差。基准附带数据格式和任务注册 SDK,研究者可以在自己的基础设施上运行。
CAE-Bench: Benchmarking AI Agents for Reusable CAE Solver Development
We developed CAE-Bench, a benchmark to evaluate AI agents on Computer-Aided Engineering (CAE) solver development. A CAE solver is more than a PDE solver; it must be able to generalize across various inputs, such as different boundary conditions, geometries, and material properties. CAE-Bench evaluates the generalizability of AI-developed CAE solvers by providing the same task, such as a heat transfer problem, multiple cases, with each case specifying an input. The AI agent develops a solver for a fixed task using only a few public cases, and the resulting solver is then tested on held-out cases, i.e., cases it has never seen before. We evaluate 12 models across a range of reasoning settings on 80 computational tasks. Our experiments show that even if a solver developed by an AI agent performs well on the public cases, it may even get 0 score on held-out cases. For tasks such as fluid dynamics problems, even the solver developed by GPT-6 Astra performs poorly on held-out cases. A data format and an SDK for task registration are provided, so the researcher can use the benchmark on their own infrastructure. We hope that CAE-Bench will help the community to develop AI agents that can develop reusable CAE solvers that generalize to various inputs.
This report is categorized under AITOP's "Papers" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows and US-China foundation model benchmarks and commercialization comparison. Key entities and technical topics under continuous tracking include CAE-Bench, GPT-6 Astra, agents, benchmarks.
Source attribution: curated from arXiv cs.AI (@Zhiyu Zhou, Tianyou Liu, Xinping Yang, Yiyao Liu, Jiayi Dong, Yihan Lin, Haokai Ye, Yifan Lu, Zeyu Dong, Yansong Li, Shengqi Zhang). Cross-source corroboration: 6 independent sources have covered this development (including IT之家, OpenAI, HIT Consultant), confirming industry-wide signal strength.