多模态数据解耦是AI理解复杂场景的关键,但双模态瓶颈长期未解。做多模态学习、跨模态推理的团队可以直接用RePercENT的即插即用架构,无需重新训练基础模型,值得关注。
多模态数据解耦表示学习能分离共享与独特特征,但现有方法局限于双模态场景。RePercENT 提出自监督框架,通过即插即用架构直接处理预提取嵌入,无需联合预训练,支持任意数量模态。该框架引入联合优化目标同时学习共享与独特成分,并提供理论最优性保证。实验表明,RePercENT 在多种模态和任务中成功恢复解耦成分,性能持平且计算复杂度显著降低。这为多模态AI系统(如自动驾驶、医疗诊断)提供了可扩展的基础方案。
RePercENT: Scaling Disentangled Representation Learning Beyond Two Modalities
To leverage the full potential of multimodal data, we need representations that go beyond the state-of-the-art alignment and fusion approaches and exploit all cross-modal interactions without sacrificing modality-specific information. Learning disentangled representations is a principled way to identify these underlying shared and unique factors that are hidden in observational data. However, while multimodal disentanglement is a compelling paradigm, existing methods are largely confined to the two-modality regime due to its inherent scalability bottleneck. To address this, we propose RePercENT, a self-supervised framework designed to surpass these limitations and unlocks scalable pairwise disentanglement beyond two modalities. Through a multimodal `plug-and-play' architecture, our approach operates directly on pre-extracted embeddings, eliminating the need for extensive joint pre-training while making no assumptions regarding the underlying modalities or foundation model backbones. Moreover, we introduce a joint optimization objective for simultaneously deriving the shared and unique components, and provide formal theoretical guarantees that characterize the optimality of our solution. Across diverse modalities and tasks, RePercENT successfully recovers disentangled components while maintaining competitive performance and significantly reducing computational complexity.