机器人控制不用再扛百亿参数VLM了,用ViT密集补丁特征直接训练,速度更快、参数量少99.3%,性能还涨18%。想轻量高效做具身策略的可以看这篇。
Patch Policy是一种轻量级架构扩展,让基于Transformer的机器人策略直接使用预训练的ViT密集补丁特征,避免全量视觉语言模型(VLM)的计算开销。该方法引入块因果注意力掩码,在保持时间因果性的同时处理多个补丁令牌。在4个仿真和3个真实环境套件中,Patch Policy相比使用全局池化表示的策略实现了40%的相对提升。此外,它仅用OpenVLA-OFT约0.7%的参数,就超越了微调后的OpenVLA-OFT 18%的性能。
Patch Policy: Efficient Embodied Control via Dense Visual Representations
Pretrained dense visual features from Vision Transformers (ViTs) are powerful yet have been underutilized in robot learning. Modern robot policies either compress each observation into a single global token, or rely on visual backbones trained from scratch, sacrificing both fine-grained spatial detail and the benefits of large-scale visual pre-training. While there exist policies that do operate on dense patch features like large vision-language-action models (VLAs), they tend to be heavy and slow, inheriting the full cost of a billion-parameter vision-language model (VLM) backbone. We close this gap with Patch Policy, a minimal architectural extension that enables transformer-based policies to consume dense pre-trained patch tokens directly without the computational overhead of a full VLM. At its core is a block-causal attention mask that preserves the temporal causality of standard policies while letting the model attend over many patch tokens per observation, alongside other state information. Patch Policy is lightweight, fast, and highly effective. Across four simulated and three real-world environment suites, our method achieves a 40% relative improvement over policies using state-of-the-art global-pooled representations. Furthermore, it surpasses fine-tuned OpenVLA-OFT by 18% while using roughly 0.7% of the parameters. We believe Patch Policy provides a pipeline for the robotics community to readily leverage continuing progress in visual representation learning, without sacrificing the training efficiency or inference speed required for high-frequency, reactive control. Videos can be viewed at https://patch-policy.github.io