Stability AI 开源交互式视觉训练框架 IVG，评估实时指导型视觉助手
给做视觉助手和 agent 的开发者提个醒：Stability AI 开源了一个闭环仿真评测框架，自带 315 个模拟用户场景，连 Gemini-4-Argon 都只能按时完成 53.4% 的任务。想给自己的多模态助手找更真实的评测和训练环境，可以直接拿它的代码跑起来。
Stability AI 推出 Interactive Visual Gym（IVG），一个开源仿真框架，用于评估和训练能在任务过程中提供实时指导的视觉 AI 助手。框架在模拟世界中重建烹饪、家务和手机操作任务，并引入具有不同技能水平和偏好的模拟用户，他们会行动、犯错、提问并回应指导。
团队用 14 个助手跑完 315 个场景，成绩最好的模型 Gemini-4-Argon 也只有 53.4% 的任务能按时完成。模拟器的打分与人类偏好约 80% 的情况一致。用该仿真环境训练后，一个 4B 开源模型在未见过的任务上的成功率从 33.3% 提升到 42.2%。
这个框架解决的核心问题是：传统静态基准测不出视觉助手在动态、多轮交互中是否真的有用。对做 agent 和多模态助手的开发者来说，可以在 IVG 里做闭环训练与评测，代码和论文都已开放。
Interactive Visual Gym: Evaluating Visual Interaction Models Through Simulation
Can a visual AI assistant’s real-time advice actually help someone finish a procedural task? Meet Interactive Visual Gym (IVG), an open simulation framework for evaluating and training visual AI assistants in closed loop. We recreate cooking, household, and phone tasks in a simulation world alongside simulated users with different skills and preferences. These users act, make mistakes, ask questions, and respond to guidance – allowing us to test how accurately, appropriately, and promptly an assistant supports them as the activity unfolds.
→ 14 assistants, 315 scenarios: the best performing model, Gemini-4-Argon, completes only 53.4% of tasks on time → Human preferences match our simulator scores ~80% of the time → Training in the simulator lifts a 4B open model’s success rate’ from 33.3% to 42.2% on unseen tasks
Explore the paper, code, and demos: https://t.co/hV3Isv65sF
#EnterpriseAI #AgenticAI #FutureOfAI
This report is categorized under AITOP's "Papers" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows and US-China foundation model benchmarks and commercialization comparison. Key entities and technical topics under continuous tracking include Stability AI, Gemini, 多模态, 智能体.
Source attribution: curated from Salesforce AI Research (@SFResearch).
