混合情感识别是多模态 AI 的难点,这个排序感知融合框架解决了编码器选择问题,做情感计算或多模态融合的团队可以直接参考其方法。
本文提出了一种基于排序感知的选择性融合框架,用于解决混合情感识别中多模态线索重叠的挑战。该方法通过注意力门控模块估计每个编码器的重要性,仅融合最有效的 top-n 编码器,并解耦预测为存在性和显著性两个头部。在 BlEmoRE 挑战中,该框架超越了强个体编码器和朴素多编码器融合基线,最终获得第二名。这项工作展示了排序感知融合在细粒度混合情感识别中的有效性。
Ordering Matters: Rank-Aware Selective Fusion for Blended Emotion Recognition
Blended emotion recognition is challenging because emotions are often expressed as mixtures of subtle and overlapping multimodal cues rather than a single dominant signal. We propose a rank-aware multi-encoder framework that selectively combines complementary representations from diverse pre-extracted video and audio encoders. Our method projects heterogeneous encoder features into a shared latent space, estimates sample-wise encoder importance through an attention-based gating module, and fuses only the top-n most informative encoders. To better model blended emotions, we decouple prediction into presence and salience heads and align them through probability-level fusion. We further incorporate feature-level unsupervised domain adaptation without pseudo-labeling to improve robustness under distribution shift. Experiments on the BlEmoRE challenge show that the proposed framework outperforms strong individual encoders and naïve multi-encoder fusion baselines. Our final system ranked 2nd in the competition, supporting the effectiveness of rank-aware selective fusion for fine-grained blended emotion recognition.