Pika Soundtrack：用潜在扩散模型为视频生成同步音频
精选理由
Pika 发了 Soundtrack，能把视频压成潜在令牌再生成同步音频，时序和场景都对得上，做视频配乐可以试试。
Pika Soundtrack 将视频压缩为保留时序、运动和场景布局的潜在令牌，提示词则转化为指定应听到内容的语义令牌。一个潜在扩散 Transformer 在去噪音频时对两者进行交叉注意力处理，从而学习场景中应有什么声音以及何时发生。该方法旨在让生成的音频与画面内容及节奏精准同步。
原文 · Pika Labs
Pika Soundtrack is architected to compress video into latent tokens that preserve timing, motion, and scene layout. The prompt becomes semantic tokens that specify what should be heard. A latent diffusion transformer cross-attends to both while denoising the audio—so it learns both what belongs in the scene and when it should happen. 💬 1 🔄 0 ❤️ 1 👀 477 📊 1 ⚡