LeVo 2:分层建模与渐进后训练实现稳定歌曲生成

LeVo 2: Stable and Melodious Song Generation via Hierarchical Representation Modeling and Progressive Post-Training

精选理由

LeVo 2 发布了一个能生成完整歌曲的模型,通过分层建模和美学引导训练,歌唱质量和可控性都比开源方案强,已经逼近商业系统水平。

AI 摘要

LeVo 2 是一个混合 LLM-Diffusion 框架,用于可控完整歌曲生成。它通过分层建模,先用 LeLM 预测混合 token 进行语义规划,再并行预测声乐和伴奏 token 细化细节,最后用扩散 Music Codec 重建波形。该框架引入美学引导训练,前训练阶段用自动音乐评估为大规模数据标注音乐性层级。渐进后训练依次使用 SFT、大规模离线 DPO 和闭环半在线 DPO 提升生成质量、可控性和音乐性。专家听测和客观评估显示,LeVo 2 在六个主观维度上超越开源基线,并在多项指标接近商业系统。

原文 · arXiv cs.AI

LeVo 2: Stable and Melodious Song Generation via Hierarchical Representation Modeling and Progressive Post-Training

Full-length song generation must preserve coherence and musicality, render detailed vocal and accompaniment acoustics, and follow lyrics and prompts. Existing language model-based systems face a structural trade-off: mixed-token modeling preserves vocal-instrument coordination but obscures track-specific details, whereas dual-track prediction improves acoustics but requires longer sequences and weakens global planning. We present LeVo 2, a hybrid LLM-Diffusion framework for controllable full-length song generation. LeVo 2 formulates this trade-off as hierarchical modeling: LeLM first predicts mixed tokens for semantic planning, then predicts vocal and accompaniment tokens in parallel for track-specific refinement, while a diffusion-based Music Codec reconstructs full-length waveforms. A central contribution of this extended version is an aesthetics-guided training schedule for alignment. During pre-training, an automated music aesthetic evaluation framework assigns musicality-tier conditions to large-scale data, providing musicality priors before preference alignment. Progressive post-training applies SFT, large-scale offline DPO, and closed-loop semi-online DPO to separately improve generation quality, controllability, and musicality. Modular extension then trains the Track-Specific LM for acoustic refinement while preserving the aligned semantic planner. This schedule separates musicality learning, controllability alignment, and acoustic refinement, mitigating optimization conflict and the limitations of static offline preference pairs. Expert listening tests and objective evaluations show that LeVo 2 outperforms open-source baselines across six subjective dimensions, and approaches leading commercial systems on several listening metrics. Ablations validate the effects of the training strategy, aesthetics guidance, scaling, and hierarchical architecture.