精选理由
Pika 发了 Soundtrack，能把视频压成潜在令牌再生成同步音频，时序和场景都对得上，做视频配乐可以试试。
Pika Soundtrack 将视频压缩为保留时序、运动和场景布局的潜在令牌，提示词则转化为指定应听到内容的语义令牌。一个潜在扩散 Transformer 在去噪音频时对两者进行交叉注意力处理，从而学习场景中应有什么声音以及何时发生。该方法旨在让生成的音频与画面内容及节奏精准同步。
原文 · Pika Labs
Pika Soundtrack is architected to compress video into latent tokens that preserve timing, motion, an...
Pika Soundtrack is architected to compress video into latent tokens that preserve timing, motion, and scene layout. The prompt becomes semantic tokens that specify what should be heard. A latent diffusion transformer cross-attends to both while denoising the audio—so it learns both what belongs in the scene and when it should happen. 💬 1 🔄 0 ❤️ 1 👀 477 📊 1 ⚡