风格字幕TTS中指令如何塑造语音?交叉注意力归因研究

How Do Instructions Shape Speech? Cross-Attention Attribution for Style-Captioned Text-to-Speech

精选理由

这篇把视觉归因方法搬到语音合成里,让你看清每个词怎么影响声音,对理解可控TTS特别有用。

AI 摘要

论文首次将DAAM框架适配到语音扩散模型,提出交叉注意力归因方法并应用于CapSpeech-TTS。方法提取了25层和24个ODE步骤的每个token热力图。分析了3600个(风格字幕,文本转录)组合,包含120个风格字幕和30个文本转录。结果发现风格token的时间方差低于内容/功能token,且风格关注度与F0和能量相关。注意力熵在层17达到最小值,与风格重要性峰值同时出现。

原文 · arXiv cs.AI

How Do Instructions Shape Speech? Cross-Attention Attribution for Style-Captioned Text-to-Speech

Style-captioned text-to-speech systems use natural language to control voice characteristics, but how individual words influence acoustic output remains unclear. Understanding this is critical for diagnosing failure modes and improving controllability in expressive TTS. We propose cross-attention attribution for speech diffusion models, adapting the DAAM framework to the speech domain for the first time, and apply it to CapSpeech-TTS. Our method extracts per-token heatmaps across 25 layers and 24 ODE steps. We analyze 3,600 (style caption, text transcript) combinations comprising 120 style captions conditioning the generation of 30 text transcripts each, revealing how caption tokens shape waveforms. Results show: (1) style tokens have lower temporal variance than content/function tokens, confirming global conditioning; (2) style attention correlates with F0 and energy; (3) style conditioning peaks in early steps and deep layers; (4) attention entropy reaches its minimum at layer 17, co-occurring with the style importance peak, indicating maximal network selectivity at the most style-critical stage. This is the first study of how natural language influences cross-attention in speech diffusion models