音频语言模型的少样本学习:阶段调制声学提示

Acoustic Prompting via Stage-wise Modulation for Few-Shot Learning in Audio Language Models

精选理由

这篇论文把提示学习从文本拓展到音频端,在11个数据集上验证了效果,代码已开源做少样本音频分类的可以看看。

AI 摘要

提出在音频编码器中引入可训练提示(acoustic prompt)以捕获任务特定声学特征,与现有文本端提示学习结合,增强少样本适应能力。在11个数据集上的实验表明,该方法作为即插即用模块可普遍提升性能。显式调制音频表示空间有效补充纯文本提示方法。代码已开源。

原文 · arXiv cs.LG

Acoustic Prompting via Stage-wise Modulation for Few-Shot Learning in Audio Language Models

Audio-Language Models (ALMs) have shown remarkable success in zero-shot audio classification by aligning audio waveforms with text. Recent efforts to improve downstream performance focus on learning optimal text prompts. However, previous approaches focus on the text encoder, leaving the potential of learnable prompts within the audio encoder unexplored. In this paper, we propose a novel framework that introduces trainable prompts into the audio encoder to capture task-specific acoustic features. We demonstrate that integrating audio-side prompt learning with existing text-side approaches enhances few-shot adaptation. Through extensive experiments across 11 datasets show that integrating our method as a plug-and-play module alongside existing text prompt tuning generally leads to performance improvements. These findings suggest that explicitly modulating the audio representation space effectively complements text-only prompting approaches. The code is available at https://github.com/hyebin-c/aspl.