EgoGenesis:结合在线锚定投影记忆与Action-3D RoPE的自我中心世界动作建模

EgoGenesis: Egocentric World-Action Modeling with Online Anchored Projective Memory and Action-3D RoPE

精选理由

EgoGenesis是个能生成机器人操作视频的工具,用400条合成数据就把单臂成功率从77%提到84%,双臂从53%提到70%,挺厉害。

AI 摘要

EgoGenesis是一个基于预训练视频生成先验的自我中心世界动作模拟器,用于合成可控的操作视频。它通过在线锚定投影记忆(OAPM)保留第一帧3D场景锚点,并用Action-3D RoPE编码末端执行器运动。将400条真实轨迹与400条EgoGenesis生成轨迹合并后,单臂任务真实机器人成功率从77%提升至84%。双臂任务成功率从53%提升至70%,说明合成数据能显著改善下游世界动作模型的泛化。

原文 · arXiv cs.AI

EgoGenesis: Egocentric World-Action Modeling with Online Anchored Projective Memory and Action-3D RoPE

Egocentric video offers rich manipulation experience for embodied AI, yet collecting diverse egocentric data across scenes, objects, motions, and embodiments remains costly. We present \method, an egocentric world-action simulator that synthesizes controllable, high-quality manipulation videos to expand scarce real-world training data. \method{} builds on a pretrained video generation prior and introduces two geometry-aware conditioning mechanisms. Online Anchored Projective Memory (OAPM) preserves a first-frame 3D scene anchor while periodically refreshing a recent state during autoregressive generation. Action-3D Rotary Position Embedding (A3D-RoPE) encodes end-effector motion with camera-aware 3D rotary coordinates, injecting action geometry into skeleton-to-video cross-attention for precise control. Together, these components improve visual fidelity, geometric stability, and action alignment in long egocentric rollouts. Moreover, augmenting 400 real trajectories with 400 \method-generated trajectories improves out-of-distribution real-robot success from 77\% to 84\% on single-arm tasks and from 53\% to 70\% on dual-arm tasks, demonstrating that the synthesized data substantially improve downstream WAM generalization.