多模态模型训练中模态竞争是常见痛点，做多模态预训练或大batch训练的团队可以直接参考这个二阶优化方案，能显著提升训练效率和稳定性。
多模态自回归模型在图像生成和文本理解联合训练时，存在模态竞争导致优化不稳定、大batch训练受限的问题。研究发现AdamW等一阶优化器对跨模态梯度异质性敏感，而二阶预条件方法（如SOAP）能提供更稳定的对齐基础。作者提出ML-FOP-SOAP框架，通过Fisher正交投影抑制方差引起的模态冲突，并引入分层折叠策略降低计算开销。在Janus和Emu3模型上，该方法在batch size 8192下训练稳定，样本效率提升1.4倍，训练速度提升1.5倍。
Second-Order Multi-Level Variance Correction for Modality Competition in Multimodal Models
Autoregressive next-token training offers a unified formulation for image generation and text understanding, but it also creates strong modality competition that destabilizes optimization and limits large-batch scaling. We show that first-order optimizers such as AdamW are vulnerable to cross-modality gradient heterogeneity, while second-order preconditioning, particularly SOAP, provides a more stable basis for multimodal alignment. Building on this insight, we propose \emph{ML-FOP-SOAP}, a second-order optimization framework with Multi-Level Variance Correction. Our Fisher-Orthogonal Projection suppresses variance-induced modality conflicts, reducing the trade-off between visual generation and textual understanding. To make this practical under large gradient accumulation, we introduce a hierarchical folding strategy that captures fine-grained variance with low micro-step overhead. Experiments on Janus and Emu3 show consistent gains across both modalities and stable training at batch size 8192. Compared with AdamW, our method improves sample efficiency by up to $1.4\times$ and accelerates wall-clock training by up to $1.5\times$, offering a robust optimizer for scaling multimodal foundation models.