Papers

AdaGround:免训练 GUI 定位框架在 ScreenSpot-Pro 上提升 20.6 点

AdaGround: Token-Normalized Adaptive Zooming with Verifier-in-the-Loop Refinement for GUI Grounding

Why we picked it

免训练就能让 Qwen3-VL 在 GUI 定位基准上多拿 7 到 20 个点,做屏幕自动化的开发者可以直接看代码试试。

上海交大团队提出 AdaGround,一个免训练的 GUI 定位框架,针对高分辨率屏幕上小目标定位失败的问题。它的 Token-Normalized Adaptive Zooming 会根据模型对目标大小的估计来设定放大倍数、视野和缩放深度。Verifier-in-the-Loop Refinement 用一个标记点击 Yes/No 验证器来决定保留或拒绝缩放、触发重新定位,并在必要时弃权。在 Qwen3-VL 模型上,AdaGround 在 OSWorld-G 和 ScreenSpot-Pro 两个基准上比骨干模型提升 7.4 到 20.6 点,超过了现有免训练缩放方法和多个训练过的 GUI 模型。

Original · arXiv cs.AI

AdaGround: Token-Normalized Adaptive Zooming with Verifier-in-the-Loop Refinement for GUI Grounding

GUI grounding on high-resolution screens fails mostly on small targets, and zooming in is a common training-free remedy. However, most existing zooming pipelines do not match the zoom to the target's size, accept the zoomed answer unconditionally and never revisit a wrong answer. To address these issues, we propose AdaGround, a training-free framework with three components: (1) Token-Normalized Adaptive Zooming, which sets the magnification, field of view and zoom depth of each sample from the model's own estimate of the target size; (2) Verifier-in-the-Loop Refinement, which uses one marked-click Yes/No verifier to keep or reject each zoom, choose between zoom chains, trigger re-grounding and, with an existence query, decide when to abstain; and (3) Compositional Structure Grounding, which composes spreadsheet cells from their headers. On Qwen3-VL models, AdaGround surpasses state-of-the-art training-free zooming methods and many trained GUI models, improving accuracy over the backbone by 7.4 to 20.6 points on OSWorld-G and ScreenSpot-Pro, and its gains carry over to held-out UI-Vision. Code is available at https://github.com/bcwang-sjtu/AdaGround