Imaging-101: 针对科学计算成像的LLM编码智能体基准测试

Imaging-101: Benchmarking LLM Coding Agents on Scientific Computational Imaging

精选理由

这基准测试了LLM在科学计算成像编程上的真实水平,发现它们在算法选择和物理建模上容易出错,挺有参考价值。

AI 摘要

Imaging-101是一个包含57个专家验证的计算成像任务的基准,覆盖六个科学领域。每个任务基于同行评审论文,标准化为四个阶段(预处理、正向物理建模、逆求解、可视化)。基准设三个评估轨道(规划、函数级单元测试、端到端重建),测试了七个前沿LLM的系统性能力。结果发现LLM在算法选择、物理惯例处理和流水线集成方面存在具体短板。

原文 · arXiv cs.AI

Imaging-101: Benchmarking LLM Coding Agents on Scientific Computational Imaging

Computational imaging, which recovers hidden signals from indirect, noisy measurements, underpins quantitative discovery across scientific disciplines, yet building a correct reconstruction pipeline demands deep domain expertise and remains laborious even for domain scientists. We introduce Imaging-101, a benchmark of 57 expert-verified computational imaging tasks spanning six scientific domains, each grounded in a peer-reviewed paper and canonicalized into a standardized four-stage pipeline (preprocessing, forward physics modeling, inverse solver, and visualization) Three evaluation tracks (planning, function-level unit tests, and end-to-end reconstruction) probe distinct agent capabilities across the full pipeline. Evaluating seven frontier LLMs uncovers systematic challenges in applying coding agents to computational imaging that go beyond those exposed by general coding benchmarks, spanning algorithm selection, physical convention handling, and pipeline integration. These findings highlight concrete capability gaps and point toward skill-augmented, domain-specialized agents as a practical path to reliable computational imaging assistance.