做GUI自动化智能体的团队终于有了能记住历史并看清屏幕的批评器——HiViG在长任务中显著提升成功率,建议做CUA开发的直接看论文。
计算机使用智能体(CUA)在执行GUI任务时,现有批评模型存在短视和缺乏视觉基础两大局限。研究者提出HiViG框架,通过多模态批评器记录历史动作并基于截图验证执行坐标,在预执行阶段拦截错误。在网页、移动和桌面基准测试中,HiViG使Qwen3-VL-32B和Gemini-3-Flash的成功率分别提升5.8%和9.0%,并展现出强跨平台泛化能力。消融实验表明,宏观动作历史和视觉基础批评对长程GUI任务至关重要。
A History-Aware Visually Grounded Critic for Computer Use Agents
Various test-time interventions for Computer Use Agents (CUAs), including critic models, have been developed to improve performance through pre-execution action evaluation in complex Graphical User Interface (GUI) environments. However, existing critics suffer from two key limitations: they (1) focus primarily on short-sighted decision loops (e.g., forgetting earlier actions) and (2) lack the visual grounding needed to detect flawed actions (e.g., clicking wrong UI elements). To address these, we introduce HiViG, a History-aware Visually Grounded test-time framework, built around a multimodal critic trained on real GUI trajectories to abstract past interactions into a compact record and to evaluate actions with visual grounding. At test time, HiViG integrates the critic into the policy decision loop to provide macro-action history, which summarizes the policy's completed achievements, and visually grounded critique, which verifies raw execution coordinates against the current screenshot to intercept errors before execution. Across web, mobile, and desktop benchmarks, HiViG consistently outperforms existing scalar and verbal critics, improving average success rates over the strongest baseline by 5.8% for Qwen3-VL-32B and 9.0% for Gemini-3-Flash, and demonstrates strong cross-platform generalization. Ablations show that macro-action history mitigates short-sighted planning and visually grounded critique reduces execution errors, with both components being critical for test-time scaling in long-horizon GUI tasks.