PapersMulti-source

CAE-Bench 发布:评估 AI 智能体开发可复用 CAE 求解器的能力

CAE-Bench: Benchmarking AI Agents for Reusable CAE Solver Development

Why we picked it

搞 CAE 仿真的工程团队和做智能体评测的研究者应该看看。它专挑求解器泛化能力下手,80 个任务、12 个模型测下来,连 GPT-6 Astra 在流体任务上都翻车,比通用代码基准更能暴露智能体的真实短板。附带了数据格式和 SDK,可以直接接进自己的测试环境。

研究者发布 CAE-Bench,一个用于评估 AI 智能体开发计算机辅助工程(CAE)求解器能力的基准。CAE 求解器不只是 PDE 求解器,还需要在不同边界条件、几何形状和材料属性等输入上保持泛化。基准的设计方式是:同一任务(如传热问题)提供多个用例,智能体只能使用少量公开用例开发求解器,然后在从未见过的保留用例上测试。

Body

实验覆盖 12 个模型、80 个计算任务和多种推理设置。结果显示,AI 开发的求解器即使公开用例上表现良好,在保留用例上也可能得 0 分;在流体动力学任务上,连 GPT-6 Astra 开发的求解器在保留用例上也表现很差。基准附带数据格式和任务注册 SDK,研究者可以在自己的基础设施上运行。

Original · arXiv cs.AI

CAE-Bench: Benchmarking AI Agents for Reusable CAE Solver Development

We developed CAE-Bench, a benchmark to evaluate AI agents on Computer-Aided Engineering (CAE) solver development. A CAE solver is more than a PDE solver; it must be able to generalize across various inputs, such as different boundary conditions, geometries, and material properties. CAE-Bench evaluates the generalizability of AI-developed CAE solvers by providing the same task, such as a heat transfer problem, multiple cases, with each case specifying an input. The AI agent develops a solver for a fixed task using only a few public cases, and the resulting solver is then tested on held-out cases, i.e., cases it has never seen before. We evaluate 12 models across a range of reasoning settings on 80 computational tasks. Our experiments show that even if a solver developed by an AI agent performs well on the public cases, it may even get 0 score on held-out cases. For tasks such as fluid dynamics problems, even the solver developed by GPT-6 Astra performs poorly on held-out cases. A data format and an SDK for task registration are provided, so the researcher can use the benchmark on their own infrastructure. We hope that CAE-Bench will help the community to develop AI agents that can develop reusable CAE solvers that generalize to various inputs.

Industry Coordinates & Source Verification

This report is categorized under AITOP's "Papers" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows and US-China foundation model benchmarks and commercialization comparison. Key entities and technical topics under continuous tracking include CAE-Bench, GPT-6 Astra, agents, benchmarks.

Source attribution: curated from arXiv cs.AI (@Zhiyu Zhou, Tianyou Liu, Xinping Yang, Yiyao Liu, Jiayi Dong, Yihan Lin, Haokai Ye, Yifan Lu, Zeyu Dong, Yansong Li, Shengqi Zhang). Cross-source corroboration: 6 independent sources have covered this development (including IT之家, OpenAI, HIT Consultant), confirming industry-wide signal strength.