这篇论文提出用时间差异做视频生成音频，不用额外网络，效果还超过了对比预训练方法，值得做V2A的人看看。
TD-V2A是一种基于扩散模型的视频到音频生成方法，利用时间差异作为区分视频与图像的关键表示。研究对比了帧级和特征级的时间差异，发现特征级更有效。该方法引入分层持续学习策略和退火时间差异引导，在基准数据集上取得优于对比音视频预训练的效果。
Visual Representation Matters: Exploiting Temporal Differences in Video-to-Audio Generation
Video-to-audio (V2A) generation extends image-to-audio generation (I2A) by introducing consecutive frames that provide essential temporal cues for audio synthesis. However, existing conditional diffusion-based V2A methods typically enhance visual conditioning with additional audio-visual supervision, acoustic structure prediction, or reasoning from large multimodal models, requiring extra networks or strong inductive biases. Inspired by recent advances in visual representation learning, we introduce TD-V2A, which leverages temporal differences (TD) as the key representation that distinguishes V2A from I2A, enriching visual conditioning with minimal architectural modification. We first investigate TD at both the frame and feature levels to identify the most effective representation level at which TD complements visual representations. Based on these findings, we develop a hierarchically continual learning strategy and an annealed temporal differences guidance method to progressively learn and exploit TD information during diffusion training and sampling process, respectively. Extensive experiments on benchmark datasets demonstrate that effectively exploiting TD through our proposed framework significantly improves end-to-end V2A generation quality, even outperforming dedicated V2A representations such as contrastive audio-visual pretraining.