手术机器人数据难搞,这个模型用便宜的视频预训练把成功率拉高了 14 个点,PegTransfer 直接涨 20%,搞机器人学习的可以看看。
Surgical WAM 是一个基于 Cosmos Policy 构建的统一生成模型,能同时预测内窥镜观察和可执行的手术机器人动作块。它先从无动作视频中学习手术视觉动态,再在固定数量的动作标注数据上微调。在四项模拟手术任务中,视频预训练将平均成功率从 63.5% 提升至 77.8%,其中 PegTransfer 任务绝对提升 20 个百分点。接触密集和双臂协调任务受益最大,表明无动作视频可提供可迁移的视觉动态先验。
Surgical WAM: A World-Action Model for Data-Efficient Surgical Robot Learning
Learning reliable surgical manipulation policies is bottlenecked by the scarcity of action-labeled demonstrations: teleoperated surgical robot (e.g., dVRK) trajectories with synchronized kinematics are costly to collect, while surgical tasks demand precise contact handling, long-horizon reasoning, and bimanual coordination. Endoscopic video is comparatively inexpensive and abundant relative to synchronized video--kinematics trajectories, and a natural way to exploit it is to learn world models of surgical scenes. However, existing surgical world models use video primarily for simulation or policy evaluation, and rarely translate the learned dynamics into closed-loop control. This gap raises our central question: under a fixed budget of action-labeled demonstrations, does action-free video pretraining improve closed-loop surgical manipulation? To answer it, we introduce the Surgical World-Action Model (Surgical WAM), a unified generative model built on Cosmos Policy that jointly predicts future endoscopic observations and executable surgical robot action chunks. Surgical WAM first learns surgical visual dynamics from action-free video and is then fine-tuned on the fixed action-labeled budget; at deployment, it acts as a closed-loop, receding-horizon controller that executes a short prefix of each predicted action chunk and replans from the resulting observation. On a suite of four simulated surgical manipulation tasks, video pretraining improves the average success rate from 63.5% to 77.8%, including an absolute gain of 20 percentage points on PegTransfer, with the largest improvements on contact-rich and bimanual tasks. These results demonstrate that action-free video provides transferable visual dynamics priors for learning surgical robot control with limited action supervision, positioning data-efficient video pretraining as a practical path toward scaling up surgical robot learning.