GUI智能体任务规划:自主经验探索与事后经验利用

Empowering GUI Agents via Autonomous Experience Exploration and Hindsight Experience Utilization for Task Planning

精选理由

这篇让7B小模型在GUI任务规划上超过32B大模型,还分析了不同层级任务泛化的差异,很实用。

AI 摘要

该论文提出PEEU方法,通过自主环境探索发现经验并利用事后经验生成严格对齐的高层训练数据。7B模型在真实基准上达到30.6%准确率,超越Qwen2.5-VL-32B。作者提出TDHAF框架分析任务分解的组成性泛化,发现低层原子技能掌握不保证高层规划能力,而高层任务训练对OOD泛化更关键。

原文 · arXiv cs.LG

Empowering GUI Agents via Autonomous Experience Exploration and Hindsight Experience Utilization for Task Planning

Multimodal web agents can assist humans in operating repetitive GUI tasks, where effective task planning is essential for decomposing complex tasks into executable actions. While small open source MLLMs are cost efficient and privacy preserving compared with commercial large models, they suffer from weak planning and limited cross website generalization. To address these limitations, we introduce the planning experience exploration and utilization (PEEU) method, which autonomously explores environments to discover experiences and utilizes hindsight experience to synthesize strictly aligned, high level training data. To quantitatively analyze the generalization behaviors driving this performance, we propose the task decomposition hierarchical analysis framework (TDHAF) to systematically study compositional generalization across three task granularities: low, middle and high levels. Our analysis reveals that mastering low level atomic skills does not guarantee high level planning competence, while high level task training yields stronger OOD generalization. Experiments on real world benchmarks demonstrate PEEU's superior effectiveness: our 7B model achieves 30.6% accuracy, outperforming the much larger Qwen2.5-VL-32B model. These demonstrate constructing hindsight high level tasks and leveraging experiences is crucial for OOD planning abilities of small MLLMs.