Lumos-Nexus:高效频率桥接实现视频统一模型

Lumos-Nexus: Efficient Frequency Bridging with Homogeneous Latent Space for Video Unified Models

精选理由

视频生成领域终于有了兼顾推理能力和视觉保真度的方案,做视频理解与生成统一模型的团队可以直接参考其两阶段设计,省去大量训练成本。

AI 摘要

Lumos-Nexus 是一种训练高效的统一视频生成框架,解决了将高保真生成器集成到统一训练循环中计算成本过高的问题。它采用两阶段设计:训练时仅用轻量级生成器与理解模块对齐,学习推理驱动的语义控制;推理时通过统一渐进频率桥接(UPFB)在共享潜在空间中将生成任务逐步交给高容量预训练生成器,实现从粗到细的优化,生成高保真视频而不牺牲推理质量。为填补推理驱动视频生成基准的空白,团队引入了 VR-Bench 评估模型将推断意图转化为连贯视频的能力。实验表明,Lumos-Nexus 在 VBench 上显著提升了视觉真实感和时间连贯性,在 VR-Bench 上展示了强大的推理生成性能。代码和模型已开源。

原文 · arXiv cs.AI

Lumos-Nexus: Efficient Frequency Bridging with Homogeneous Latent Space for Video Unified Models

Connector-based video unified models have demonstrated strong capability in instruction-grounded video synthesis, but integrating a large high-fidelity generator into the unified training loop is computationally prohibitive, limiting achievable visual quality. We therefore propose Lumos-Nexus, a training-efficient unified video generation framework that facilitates the development of strong reasoning-driven generation capabilities while significantly enhancing visual fidelity. Lumos-Nexus adopts a two-stage design: 1) During training, only a lightweight generator is aligned with the understanding block to learn to take in reasoning-driven semantic control. 2) During inference, we introduce Unified Progressive Frequency Bridging (UPFB) to progressively hand off generation to a high-capacity pretrained generator in the shared latent space, enabling coarse-to-fine refinement and producing high-fidelity videos without compromising reasoning quality. To fill the gap in reasoning-driven video generation benchmarks, we introduce VR-Bench, which assesses a model's capability to translate inferred intent into coherent and semantically aligned video content. Extensive experiments demonstrate that Lumos-Nexus achieves substantial gains in visual realism and temporal coherence on VBench, while exhibiting strong reasoning-based generative performance on VR-Bench. Code and models are available at https://jiazheng-xing.github.io/nexus-lumos-home/.