做机器人操作和感知的团队,DynaFLIP 把运动理解前移到预训练阶段,直接提升下游策略泛化性,分布外场景提升 22.5% 值得关注。
DynaFLIP 提出一种动力学感知的多模态预训练框架,通过构建图像-语言-3D 流三元组,将运动理解从下游策略上提到感知层。其核心创新是让三种模态在共享超球面空间中形成小单纯形体积,结合体积最小化、余弦正则化和对比学习目标,避免几何歧义和塌缩。实验表明,DynaFLIP 能聚焦于操作相关的控制区域,作为可复用的视觉骨干,在多种下游策略(包括 VLA)上持续超越基线,在分布外场景下性能提升达 22.5%。这项工作表明,训练视觉表征不仅编码静态内容,还编码动作下的世界变化,能显著提升机器人泛化能力。
DynaFLIP: Rethinking Robotics Perception via Tri-Modal-Dynamics Guided Representation
Robot manipulation critically depends on perception that preserves the action-relevant aspects of a scene. Yet most robot learning pipelines are built upon visual encoders pre-trained for static recognition or vision-language alignment, leaving motion understanding to downstream policies. We introduce DynaFLIP, a dynamics-aware multimodal pre-training framework that pushes motion understanding upstream into perception. We construct image-language-3D flow triplets from heterogeneous human and robot videos, and use these triplets as training-time supervision to shape an image-only encoder. Our key idea is to encourage the three modalities to span a small simplex volume in the shared hyperspherical space -- a smaller simplex volume indicating stronger alignment. To avoid the geometric ambiguity and trivial collapse of naive volume minimization, we combine simplex-volume minimization with a cosine regularizer and a contrastive objective. Our analyses show that DynaFLIP focuses on control-relevant regions critical for manipulation. The resulting dynamics-aware representations serve as reusable visual backbones and consistently outperform baselines across diverse downstream policies, including VLAs. We validate this across diverse simulation and real-world setups, with gains reaching +22.5% under out-of-distribution scenarios. Our results suggest that robot generalization improves when visual representations are trained to encode not just what is present, but how the world changes under action.