论文精选

单次运行报告误导:35B计算机使用智能体修复策略的统计可靠性分析

Teach it to stop, not just to click

精选理由

这篇论文用严格的统计方法告诉你:CUA的修复效果不能只看单次跑分,一个看起来不错的改进有三分之一可能是错的。做AI agent的都应该看看这个库。

AI 摘要

使用verifier-guided repair对35B参数的计算机使用智能体(CUA)在五个oracle-graded环境中进行测试,方差分量分解显示评估方差几乎为零,训练种子效应不超过10%,而数据抽取在最难任务中占48%的主导方差。该任务的运行间分布为双模(Hartigan dip p=0.07, k=10),单次运行约有30%概率落入失败模式,均值±标准差无法正确描述。修复能力分两层:固定token安装可靠(成功率0.97±0.06),而开放式坐标点击修复仅部分有效(0.53±0.35),生成字段填充更弱(0.14±0.04)。框架级修复仅在任务是唯一剩余障碍时提升成功率(LinkedIn 8/20 vs 基准0/15, Fisher p=0.006)。作者发现单次改进报告约三分之一概率符号错误,并发布了cua_reliability库以支持多种子常规报告。

原文 · arXiv cs.AI

Teach it to stop, not just to click

Agentic computer-use RL is reported in single runs, and those numbers mislead. Using verifier-guided repair of a 35B computer-use agent (CUA) across five oracle-graded environments, we show a repaired policy's success rate is dominated by upstream variance: a variance-components decomposition across three cells (crossed data-draw $\times$ seed grid, bootstrap CIs) finds evaluation variance negligible ($σ_{\mathrm{eval}} \approx 0$) and the training-seed effect small everywhere ($\leq 10\%$); instead it splits between the data draw and run-to-run nondeterminism, the data draw's share rising to dominant ($48\%$) on the hardest cell. There the run-to-run distribution is bimodal (Hartigan dip $p=0.07$, $k=10$), so a single run has roughly a 30% chance of the failure mode and mean$\pm$std is the wrong summary. On that footing, two findings hold. First, repairability is two-tier in how constrained the corrective action is: a single fixed token installs reliably (done-detection $0.97\pm0.06$), while open-ended corrections are only partial -- spatial-coordinate clicks (grounding $0.53\pm0.35$) and a generative field-fill ($0.14\pm0.04$). Second, the frame-level repair transfers to task success only when the corrective action is the task's sole remaining blocker (LinkedIn 8/20 vs. base 0/15, Fisher $p=0.006$). We caught two of our own over-claims -- a sample-efficiency curve and a 'grounding cannot be bought' boundary -- only by replicating across seeds; a stress test makes the stakes external: a single-run improvement of the size this field publishes would have the wrong sign roughly one-third of the time in a comparable regime. We release a library (cua_reliability) for routine k-seed reporting. The apparatus is, to our knowledge, the first multimodal segment-aggregated on-policy self-distillation (SA-OPSD) update on a real 35B CUA policy.