做呼吸疾病音频诊断或短音频分类的团队，可以直接借鉴CoughSense的主动帧池化和域自适应方案，解决Whisper在短信号上的静音稀释痛点。
CoughSense 是一个利用智能手机录音自动分类五种呼吸系统疾病（健康、COVID-19、哮喘、支气管炎、肺炎）的系统。它基于 OpenAI Whisper 编码器微调，并引入主动帧 QKV 注意力池化（Active-frame QKV attention pooling），解决了 Whisper 30秒输入窗口内短咳嗽信号被静音稀释的问题。系统还通过加权采样、数据增强、平衡混合、对比学习、症状条件化和域自适应等技术，处理了19:1的类别不平衡和跨数据集域偏移。在四个公开数据集共18,301条录音上，CoughSense（Whisper-tiny，8.6M参数）达到82.3%的平衡准确率，双编码器融合版本达到85.4%。主动帧池化是最大的单一贡献模块（提升5.1个点），对任何使用Whisper的短音频任务都有参考价值。
CoughSense: Five-Class Respiratory Disease Classification via Whisper Encoder Fine-Tuning and Dual-Encoder Cross-Attention Fusion with Balanced Contrastive Learning
Automated cough analysis offers a path to low-cost respiratory screening, but most existing work stops at binary COVID-19 detection. A practical tool needs to tell apart several respiratory conditions from one cough recording on a consumer smartphone. We present CoughSense, a system that sorts cough recordings into five classes. These are healthy, COVID-19, asthma or respiratory condition, bronchitis, and pneumonia. We aggregated 18,301 recordings from four public datasets (Coswara, CoughVID, Virufy, and the West China Hospital Pediatric Cough Dataset) and used the OpenAI Whisper encoder as a pretrained backbone for cough disease classification. The main contribution is active-frame QKV attention pooling, which restricts attention to the first 200 of 1500 encoder tokens. This avoids the silence-dilution problem that arises because a 3-second cough fills only 150 tokens of Whisper's 30-second input window. Other training parts handle the 19 to 1 class imbalance and the four-dataset domain shift. These include WeightedRandomSampler, SpecAugment, Balanced Mixup with forced minority pairing, a supervised contrastive auxiliary loss, FiLM symptom conditioning, and gradient-reversal domain adaptation. A dual-encoder model fuses Whisper with the OPERA-CT respiratory foundation model through cross-attention. CoughSense (Whisper-tiny, 8.6M parameters) reached 82.3 percent balanced accuracy on five-fold cross-validation (macro-F1 of 0.817, AUC of 0.941). It beat an ImageNet-pretrained EfficientNet-B2 by 11.1 points and a ViT trained from scratch by 29.6 points. All five classes passed 74 percent recall and four of five passed 80 percent. The dual-encoder model reached 85.4 percent balanced accuracy. Active-frame pooling is the largest single contributor across all ablation components at 5.1 points, which should help any short-audio task using Whisper as a backbone.
- AI Will06-01 09:39原文