这篇论文用严谨的实验方法分析了Codex和Claude Code在不同推理努力下的表现和成本，做AI研究的可以看看怎么评估智能体能力。
该论文提出一种实验设计框架，系统评估大型语言模型编程智能体（如Codex和Claude Code）的自主模型发现行为。作者控制实验因素包括推理努力程度、任务、优化指标和训练数据组成，并针对输出质量、美元成本、时钟时间和过程复杂性等响应进行回归分析。通过效用对齐的典型分解，量化推理努力效果的主导方向是否与性能-成本效用方向一致。研究在联网词汇游戏测试床上验证，揭示了推理努力与成本及过程复杂性的洞察。
An Experimental Design Approach to Evaluating Agentic AI's Autonomous Model Discovery
Large language model coding agents increasingly perform open-ended data modeling and analysis. These agents are stochastic and adaptive, and therefore their autonomous model discovery behavior cannot be adequately characterized by a single benchmark run. In this work, we propose an experimental design and analysis framework for systematically evaluating this discovery process, quantifying its variability, and identifying important factors. The proposed framework treats these agents as stochastic model-discovery operators, which map task-specific discovery data and an optimization target to a fitted model. Specifically, we investigate two such operators, Codex and Claude Code, under controlled experimental factors including agent's reasoning effort, task, optimization metric, and composition of training data. For each agent-task-metric combination, regression models and inference are conducted for multiple responses such as output quality, dollar cost, wall-clock time, and process complexity. Furthermore, we develop a utility-aligned canonical decomposition to characterize the dominant direction of the reasoning-effort effect and to assess whether that direction aligns with a performance-cost utility direction. The proposed framework is demonstrated on a testbed of networked word-forming games with insightful findings on reasoning effort with respect to cost and process complexity.