这篇论文发布了MacAgentBench，一个包含676个macOS桌面任务的智能体基准。它用细粒度评分发现Claude Opus 4.6配合OpenClaw能拿到73.7%的正确率，而且不同模型表面分一样但实际完成能力差很多，值得研究智能体的去看。
MacAgentBench新基准包含676个任务覆盖25个macOS应用，近60%任务需要同时操作GUI和命令行。采用确定性规则评估并引入细粒度多检查点评分。实验在3个框架和16个模型上进行，最优配置Claude Opus 4.6 on OpenClaw达到73.7% Pass@1，优势主要来自技能库而非框架设计。细粒度指标显示相同Pass@1的模型在子目标完成上差异显著。
MacAgentBench: Benchmarking AI Agents on Real-World macOS Desktop
Computer use agents (CUAs) have advanced rapidly in desktop automation, and a growing number of users deploy CUAs such as OpenClaw on Mac Mini for always-on automation. However, existing benchmarks, including those for macOS, evaluate agents without framework augmentation and rely on binary evaluation. As a result, they fail to capture both the framework capabilities leveraged by modern CUAs and the partial progress on long-horizon, multi-application tasks. We present MacAgentBench, a comprehensive macOS agent benchmark comprising 676 tasks across 25 applications, with nearly 60% involving both GUI and CLI interaction. The benchmark adopts deterministic rule-based evaluation and introduces fine-grained multi-checkpoint scoring with capability annotations for multi-application tasks. Experiments across three frameworks and 16 models show that the best configuration, Claude Opus 4.6 on OpenClaw, attains 73.7% Pass@1, while this advantage is primarily driven by the skill library rather than by framework design. Fine-grained metrics further reveal that models with similar Pass@1 can differ substantially in sub-goal completion. Our code and data are publicly available at https://github.com/JetAstra/MacAgentBench.