MMG2Skill:让智能体从网络指南中蒸馏出可执行技能

MMG2Skill: Can Agents Distill In-the-Wild Guides into Self-Evolving Skills?

精选理由

做智能体长任务规划的团队终于有了把网络教程变成可执行技能的方案——MMG2Skill 直接解决了指南与技能之间的鸿沟,做 GUI 自动化或游戏 AI 的开发者可以试试这个闭环框架。

AI 摘要

网络上有大量多模态、异构、嘈杂的程序性知识,但直接用于智能体执行长周期任务效果不佳。研究者提出 guide-to-skill 学习问题,并发布首个基准 MMG2Skill-Bench。他们设计的闭环框架 MMG2Skill 能将人类指南编译为可编辑技能,在执行时条件化固定视觉语言模型,并通过轨迹级根因反馈持续修正技能。在 GUI 控制、开放游戏和策略卡牌等六个 VLM 骨干上,该方法比基线提升 12.8 到 25.3 个百分点。消融实验表明,直接提示原始指南反而会降低性能,而结构化技能构建和轨迹驱动修正是关键。

原文 · arXiv cs.AI

MMG2Skill: Can Agents Distill In-the-Wild Guides into Self-Evolving Skills?

Abundant procedural knowledge on the Web holds great potential for helping agents solve long-horizon tasks. However, such knowledge is often multimodal, heterogeneous, noisy, and implicitly assumes human executors, making it difficult to use directly as the skills required by agents. To bridge the gap between human-oriented guides and agent-executable skills, we formalize this problem as guide-to-skill learning: converting in-the-wild guides into executable skills and continuously improving them from trajectories observable to the agent. To evaluate the capability of existing agents on this task, we introduce MMG2Skill-Bench, the first benchmark designed for this problem. We further propose MMG2Skill, a closed-loop framework that compiles guides into editable skills, conditions a fixed vision-language model (VLM) agent on these skills during execution, and revises the skills from trajectory-level root-cause feedback without using benchmark scores. Across GUI control, open-ended gameplay, and strategic card play with six VLM backbones, MMG2Skill consistently outperforms vanilla baseline agents in every model-domain setting, achieving macro-average gains of +12.8 to +25.3 percentage points across backbones. Ablation studies show that directly prompting agents with raw guides can degrade performance, while both structured skill construction and trajectory-driven revision are necessary for the observed improvements. On success-inferable tasks, analyzer-based early stopping further prevents late-stage performance regressions and saves 25%-53% of attempts when the success signal is properly calibrated.

MMG2Skill:让智能体从网络指南中蒸馏出可执行技能 · AI 热点