苹果推出了MM-ToolSandBox框架，帮你测测视觉智能体到底行不行，结果很扎心：最强模型成功率不到一半，而且问题出在“看图”而不是“想事”。
MM-ToolSandBox是一个面向视觉驱动工具调用智能体的基准与评估框架，提供包含500多个工具、覆盖16个应用领域的有状态执行环境，支持多图像、多轮任务。框架通过自动场景生成流程产出258个人工验证的标准场景和50个交互式UI变体。评估12个模型（从4B开源到前沿闭源）后发现，最佳模型成功率仍低于50%。失败分析表明53%的失败源于图像信息提取错误，而非规划问题；小模型主要失败在决策，大模型则失败在感知。
MM-ToolSandBox: A Unified Framework for Evaluating Visual Tool-Calling Agents
We introduce MM-ToolSandBox, a benchmark and evaluation framework for visually grounded tool-calling agents. The framework provides a stateful execution environment spanning 500+ tools across 16 application domains, supporting multi-image, multi-turn tasks where agents must ground progressively arriving visual inputs into executable tool calls while handling realistic conversational phenomena (goal revisions, error corrections, state mutations). An automated scenario generation pipeline produces diverse, visually grounded scenarios through information-flow-guided planning and multi-stage quality filtering, yielding 258 human-verified nominal scenarios and 50 variants targeting interactive UI applications. Evaluating 12 state-of-the-art models, from 4B open-weight to frontier proprietary systems, shows that current models still lack robust visual tool-calling capability: even the best model achieves below 50% success rate. Our failure analysis further reveals that visual precision, not only planning, is a primary bottleneck for capable models: 53% of failures stem from incorrect information extraction from images despite otherwise correct task workflows. A planning-to-precision crossover emerges with scale: smaller models fail at deciding what to do, while larger models fail at perceiving what they see, suggesting fundamentally different research directions for improving models at different capability levels. The framework and the benchmark are publicly available at https://github.com/apple/ml-mmtoolsandbox