这篇用AMR动态融合音视频特征,缺失模态也能准确识别说话人。在POLY-SIM上平均99%准确率,比基线高32%,很实用。
提出自适应模态路由(AMR)模块,用于多模态多语言说话人识别。AMR使用W2V-BERT 2.0音频编码器和IResNet-18人脸编码器,通过可训练路由器动态分配模态权重。在POLY-SIM 2026评估集上,系统在4个协议上的平均准确率达99.07%,比FOP基线提升32.73%。具体成绩:英语多模态99.93%,乌尔都语多模态100.00%,英语仅音频97.50%,乌尔都语仅音频98.83%。
AMR: Adaptive Modality Routing for Multimodal Polyglot Speaker Identification
Multimodal speaker identification systems face two key challenges in real-world deployment: missing modalities and language mismatch between training and testing conditions. In practical scenarios, background multi-speaker conversations, ambient noise, and overlapping speech further degrade identification accuracy. To address these challenges, we propose a multimodal polyglot speaker identification system for the POLY-SIM 2026 Grand Challenge. The system is fundamentally built upon Adaptive Modality Routing(AMR), a modality fusion module that dynamically assesses per-sample input quality and integrates modality information. Specifically, AMR employs two modality adapters to process the embeddings extracted from a linguistically robust audio encoder(W2V-BERT 2.0) and a large-scale pretrained face encoder(IResNet-18), producing modality-adapted embeddings. Based on these adapted embeddings, a trainable router estimates dynamic modality weights, which are subsequently applied to aggregate the modality-specific logits for the final prediction. To optimize this routing mechanism, we adopt a modality-aware training strategy that constructs four types of sample pairs to simulate diverse input conditions, with KL divergence serving as explicit supervision for weight assignment. Experimental results on the POLY-SIM 2026 evaluation set show that the proposed system achieves identification accuracy of 99.93%(English multimodal, P3), 100.00%(Urdu multimodal, P5), 97.50%(English audio-only, P4), and 98.83%(Urdu audio-only, P6). The average accuracy across all four protocols is 99.07%, surpassing the Fusion and Orthogonal Projection(FOP) baseline by 32.73%.