做视频生成和相机运动控制的团队终于有了一个无需配对数据就能克隆多镜头相机运动的方案——OmniDirector 用网格运动视频统一了相机表示，直接在百万级数据上训练，效果比依赖合成配对数据的方法好很多，做视频编辑和影视制作的开发者值得关注。
OmniDirector 提出了一种通用相机运动表示方法，将相机参数编码为网格运动视频，从而支持多镜头视频生成。该框架在百万级相机网格-视频对上训练，能够协调角色、动作和相机，提供导演级别的控制。它设计了一种分层提示扩展代理，通过理解信号关系系统描述相机运动和视觉内容，实现不同控制信号的和谐集成。实验表明，OmniDirector 在复杂相机运动克隆任务上表现优异，解决了现有方法依赖配对数据且性能不佳的问题。
OmniDirector: General Multi-Shot Camera Cloning without Cross-Paired Data
Cloning camera motion from reference videos is an important task in video generation, as videos provide intuitive and precise control. Existing methods either directly use parametric representations that fail to handle multi-shot generation or synthesize cross-paired data, which suffer from data scarcity, resulting in poor performance in complicated camera motion cloning. To address these issues, we introduce a general camera motion representation that encodes cameras as grid motion videos. This camera grid represents the camera parameters visually and supports the integration of diverse trajectories for multi-shot video generation. Building upon this, we propose OmniDirector, a unified framework trained on a million-scale camera grid-video pairs that coordinates characters, actions, and cameras to provide director-level control for multimodal diffusion transformers. Furthermore, we design a novel hierarchical prompt expansion agent that harmoniously integrates different control signals by systematically describing camera motion and visual content through understanding signal relationships. Extensive experiments demonstrate the superior performance and outstanding controllability of our framework. Project page: https://ymlinfeng.github.io/OmniDirector.github.io/