基于语音印象引导的伪三元组可扩展方向跟随TTS

Scalable Direction-Following TTS via Voice Impression-Guided Pseudo Triplet Construction

精选理由

NTT研究团队发布新方法,让TTS系统能根据表演方向修改语音,无需大量训练数据。

AI 摘要

研究人员提出了一种可扩展的伪三元组构建管道,用于解决方向跟随TTS的训练数据稀缺问题。该方法使用印象可控TTS模型生成受控风格变化,并通过LLM从估计的印象差异生成自然语言方向。实验表明,仅使用伪三元组即可实现稳定的说话人保持修改,结合录制数据可进一步提高方向对齐度。

原文 · arXiv cs.LG

Scalable Direction-Following TTS via Voice Impression-Guided Pseudo Triplet Construction

Voice actors often re-read the same script while modifying their delivery in response to performance directions. We study this setting as direction-following TTS, where a system generates a new utterance that reflects a given direction relative to a reference utterance while preserving speaker identity and linguistic content. A key challenge is the lack of training data capturing such relative modifications. To address this, we propose a scalable pseudo-triplet construction pipeline that generates~(reference utterance, direction text, modified utterance) triplets. It generates controlled style variations using an impression-controllable TTS model and uses an LLM to produce natural language directions from estimated impression differences. Experimental results demonstrate that pseudo-triplets alone enable stable speaker-preserving modification, and that combining pseudo and recorded data further improves direction alignment while maintaining speaker similarity. Audio examples are available on our demo page https://ntt-hilab-gensp.github.io/IS2026pseudo/