SurgKnowBench 与 SurgAgentGym：评测通用智能体的手术场景理解能力
How to "train" a surgeon? Benchmarking generalist agents in surgical scene understanding
如果你在做医疗多模态应用，这篇论文把现有 MLLM 在手术场景的短板摸得很清楚：感知任务靠工具调用能补上，但流程评估和细粒度定位还差很远。两套评测框架可以直接拿来当自家模型的压力测试。
一篇新论文提出两阶段评估框架，用于检验通用多模态大语言模型（MLLM）在手术场景理解上的真实水平。第一阶段推出诊断基准 SurgKnowBench，统一整合公开手术数据集，覆盖多种术式与任务，对专有和开源 MLLM 进行系统评测。结果显示现有模型保留部分视觉先验，但在关系与流程理解、工作流评估上表现薄弱。
第二阶段在此基础上构建了智能体环境 SurgAgentGym：模型可以在推理时对 SurgWiki 进行有界知识检索，并调用通用视觉定位工具来完成手术任务，全程无需额外的手术领域训练。核心问题是验证工具化的证据使用能否让通用智能体即时提升手术理解能力。
结论是工具调用主要在感知密集型任务上缩小差距，而细粒度定位和专家级评估仍是持续瓶颈。这为智能体化手术理解研究提供了可复用的路径，对医疗 AI 团队和手术机器人方向的研究者有直接参考价值。
How to "train" a surgeon? Benchmarking generalist agents in surgical scene understanding
Evaluating general-purpose multimodal large language models (MLLMs) for surgical scene understanding remains limited due to scarce surgical data and fragmented evaluation protocols. We address these challenges through a two-stage evaluation framework for surgical scene understanding. In Stage I, we introduce SurgKnowBench, a diagnostic benchmark that harmonizes public surgical datasets across procedures and tasks and evaluates proprietary and open-source MLLMs to map the boundary of pretrained surgical competence. SurgKnowBench reveals that current models retain partial visual priors but remain weak on relational and procedural understanding and workflow assessment. Building on this failure frontier and motivated by the scaffolded nature of surgical training, we introduce SurgAgentGym, an agentic environment where general-purpose MLLM agents can use bounded knowledge search over SurgWiki and general-purpose visual grounding for surgical tasks. SurgAgentGym evaluates whether generalist agents can improve surgical understanding at inference time through tool-mediated evidence use, without additional surgical training. Our results show that agentic inference narrows the gap mainly for perception-heavy tasks, while fine-grained localization and expert-style assessment remain persistent bottlenecks. This opens a path toward future studies of agentic surgical understanding.
This report is categorized under AITOP's "Papers" desk. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include SurgKnowBench, SurgAgentGym, 多模态, 智能体.
Source attribution: curated from arXiv cs.AI (@Gengyuan Zhang, Xiao Han, Shijie Zhou, Abdullah Koukash, Jingpei Wu, Jindong Gu, Yuqicheng Zhu, Nassir Navab, Volker Tresp, Ghazal Ghazaei).