新方法让十亿参数Transformer通过双向学习实现线性合并，损失屏障极低，视觉和语言模型都验证有效。
现有线性模式连通性方法通常只从一个模型端点优化插值路径，难以扩展到大型Transformer。我们提出新框架，应用功能保持的权重变换对齐等价解，并让两个模型双向学习向共享线性插值路径的变换。双向优化大幅减少插值障碍，在中等参数规模语言模型上实现了WikiText近零损失屏障（首次展示该规模下近无屏障线性连通）。视觉领域ViT-L在插值路径上保持ImageNet top-1准确率超69%，十亿参数LLM只表现出小损失屏障。这些结果表明解决参数对称性能使大预训练Transformer通过简单线性路径连通和合并。
Scaling Linear Mode Connectivity and Merging to Billion Parameter Pretrained Transformers
Linear mode connectivity (LMC) provides a promising foundation for understanding and merging independently trained neural networks, but existing methods typically optimize the interpolation path from only one model endpoint, limiting their scalability and effectiveness for large pretrained transformers. We propose a novel and scalable framework for enabling LMC-based model merging to {\em billion-parameter pretrained transformers}. Our method applies properly parameterized functionality-preserving weight transformations to align functionally equivalent solutions, and introduces a dual learning procedure in which both models jointly learn their corresponding transformations toward a shared linear interpolation path. This bidirectional optimization substantially reduces interpolation barriers and enables more reliable merging across large-scale architectures. Empirically, we show that our approach achieves near-zero loss barriers on WikiText for language models with medium-sized parameters, representing, to our knowledge, the first demonstration of near-barrier-free linear connectivity at this scale. In the vision domain, ViT-L maintains above 69\% ImageNet top-1 accuracy throughout the interpolation path, while modern billion-parameter LLMs exhibit only small loss barriers. These results suggest that properly resolving parameter symmetries enables large pretrained Transformers to be connected and merged through simple linear paths with substantially improved interpolation performance. Code: https://github.com/VILA-Lab/Dual-Learned-Matching .