这篇论文提出了UL4M4,一个能在半数以上模态缺失时仍保持高F1分数的无监督框架,特别适合实际中数据不全的多模态场景。
论文提出UL4M4框架,通过无监督聚类和贪心插补处理多模态学习中任意缺失模态。该方法使用模态特定归一化和部分模态距离度量,在超过50%模态缺失时仍能在F1-Micro指标上首次稳定超过0.7。框架轻量级,可适配任意融合架构,性能显著优于现有基线。
Unsupervised Learning for Missing Modalities in Multimodal Learning
This paper addresses the missing-modality challenge in multi-modal learning by introducing Unsupervised Learning for Missing Modalities in Multi-Modal Learning (UL4M4), a flexible framework that imputes missing feature embeddings in a task-independent manner before supervised prediction. We propose modality-specific normalization and a novel partial-modality distance metric to enable fair clustering of incomplete observations, capturing cross-modal structures while preserving scale-invariance across varying dimensionalities and modality counts. Cluster centers from this unsupervised stage guide an iterative greedy imputation process for any missing modalities during training or inference, supporting arbitrary numbers of modalities and arbitrary missing patterns per sample. The imputation module is lightweight, uses frozen encoders, and decouples from the downstream task, allowing easy integration with any fusion/prediction architecture. Extensive experiments under diverse and highly incomplete regimes demonstrate UL4M4's robustness, achieving, to the best of our knowledge, the first consistent F1-Micro scores above 0.7 on challenging missing configurations even when more than 50\% of modality slots are missing. Results are also stable across cluster sizes and significantly outperform state-of-the-art baselines. Code is available here: https://github.com/h-ismkhan/Multimodal-Learning-with-Missing-Modalities-via-Unsupervised-Learning.