这篇论文为多模态学习实践者提供了诊断工具，做生物医学或天体物理等异构数据研究的团队，可以在训练前判断该用对齐还是预测，避免盲目调参浪费时间。建议点开看看相图如何帮你选对目标。
该论文系统研究了多模态表示学习中两种主流范式——跨模态对齐（CA）和跨模态预测（CP）的适用条件。作者通过线性框架和信号加噪声模型，揭示了两种方法的互补失败模式：对齐在噪声相关性高时失效，预测则受源模态质量影响。他们提出了一个四区域相图（Both、CA only、CP only、Neither），并开发了数据驱动方法，帮助实践者在训练前判断应使用哪种目标。实验在合成数据、立体视觉、图像-文本对和真实天体物理数据上验证了该框架，包括跨模态训练反而有害的“Neither”区域。
When to Align, When to Predict: A Phase Diagram for Multimodal Learning
Cross-modal alignment (CA) and cross-modal prediction (CP) are the dominant paradigms for multimodal representation learning, yet there is no systematic understanding of when each succeeds, when each fails, and when cross-modal training helps at all -- a gap that leaves practitioners, especially in scientific domains like biomedicine or astrophysics, with heterogeneous instruments and multiple levels of organization and measurement, unable to diagnose why standard methods underperform the best single modality. We develop a unified linear framework that addresses both questions. Under a spiked signal-plus-noise model with structured cross-modal nuisance correlation, we derive separation ratios for both objectives that expose complementary failure modes: alignment whitens each modality and fails when nuisance is strongly correlated across views; prediction encodes whatever is cross-predictable through a one-sided whitening, with recovery governed by source-modality quality. The resulting phase diagram partitions multimodal problems into four regimes: Both, CA only, CP only, and Neither. We present a data-driven procedure to locate real-world datasets in this diagram using a small labeled subsample, identifying the preferred objective and prediction direction before any cross-modal training. Experiments on synthetic data, stereo-vision benchmarks, image-caption pairs, and real astrophysical data validate the predictions in the nonlinear regime, including the Neither regime where cross-modal training is actively harmful. Our framework lets practitioners diagnose their multimodal problem and choose the right objective before committing to training. Code to reproduce the results is available at https://github.com/IlayMalinyak/mm_align_vs_pred.