GUI Agent容易瞎点，这篇把指令理解和坐标定位拆开，用布局匹配防幻觉，ScreenSpot-Pro比端到端高20个点。
GUI代理正从依赖元数据的LLM转向基于截图的纯视觉多模态大模型。论文提出回归无关框架，冻结MLLM负责解析指令，专用定位模型通过布局先验候选匹配完成坐标定位。该方法只用文本/图标二值标签训练，不学坐标回归参数。在ScreenSpot-Pro上准确率比端到端系统提升超20%；在Mind2Web上成功率和元素选择率提升超15%。
Hallucination-Free GUI Grounding via Regression-Free Layout-Aware Matching
GUI agents are shifting from metadata-dependent large language models to purely visual multimodal large language models (MLLMs) that operate directly on screenshots. The core task, GUI grounding, requires translating abstract user instructions into precise element coordinates. This task faces a persistent dual obstacle: conventional grounding models lack the semantic richness to interpret abstract instructions, while end-to-end MLLMs suffer from coordinate hallucinations caused by deficient fine-grained perception. We propose a regression-free framework where a frozen MLLM performs instruction parsing and a dedicated grounding model handles precise localization without learning any coordinate regression. A frozen MLLM first elaborates the abstract instruction into a structured visual description rich in layout cues. These descriptions are then fed to a novel Layout-Aware GUI Grounding Model, which performs regression-free localization by matching against layout-prior candidates, inherently suppressing hallucinations and avoiding expensive fine-tuning. The grounding model is trained with only Text/Icon binary labels, requiring no coordinate regression parameters. On ScreenSpot-Pro, our method achieves over 20% improvement in grounding accuracy over end-to-end systems; on Mind2Web, it raises success rate and element selection rate by more than 15%. These results demonstrate that decoupling instruction understanding from layout-aware localization effectively resolves the core challenges of GUI interaction.