这篇论文提出了一个能生成完整歌曲的框架,支持歌词生成、纯音乐和翻唱,还在排行榜上验证了效果,音乐生成方向的同学可以看看。
本文提出一个统一的全歌曲生成框架,支持从歌词、文本描述和音乐属性生成完整歌曲。该框架包含四个组件:语义感知分词器将音频编码为8码本RVQ令牌,hybird-LM进行层次化自回归音频令牌建模,FullDiT在连续VAE潜空间中执行流匹配渲染,两级旋律模块用于翻唱生成。在人工分析音乐带歌声排行榜上取得竞争性能,并探索了DPO、GRPO和OPD作为后训练策略。实验在多语言自动基准上进行评估。
Pushing the Frontier of Full-Song Generation: Hierarchical Autoregressive Planning Meets Flow-Matching Rendering
In this report, we present a unified song generation framework capable of producing high-quality full-length music from lyrics, text descriptions, and musical attributes. The proposed framework supports three tasks: Lyrics-to-Song Generation, which generates complete songs from text descriptions, lyrics, and musical attributes; Instrumental Music Generation, which creates music without vocals; and Cover Song Generation, which reinterprets existing songs with different styles while preserving their melodic content. Architecturally, our system consists of four main components: a semantic-aware tokenizer, hybird-LM, FullDiT, and a two-level melody module. The tokenizer encodes audio into 8-codebook RVQ tokens for efficient discrete music representation. Based on these tokens, hybird-LM performs hierarchical autoregressive audio-token modeling for full-song generation. To improve audio fidelity, FullDiT performs full-song flow matching in a continuous VAE latent space conditioned on codec tokens, lyrics, and text captions. For cover song generation, the melody module extracts and discretizes melody cues from reference audio to guide generation while preserving the original melodic content. Finally, we investigate DPO, GRPO, and OPD as reward-based post-training strategies for hybird-LM and apply flow-based GRPO to FullDiT to improve musicality and rendering quality. Experimental results on a multilingual automatic benchmark, complemented by the Artificial Analysis Music with Vocals leaderboard, show that the proposed framework achieves competitive performance in the evaluated settings.