Papers

CoPre-MoE:面向多模态持续指令微调的专家合并与保留方法

CoPre-MoE: Consolidating and Preserving MoE for Multimodal Continual Instruction Tuning

Why we picked it

多模态模型连续学新任务,传统 MoE 会专家越堆越多。这篇把重复专家合并、把任务专家拆分压缩,还用冻结检索键防遗忘。做持续学习或多模态微调的工程师和研究者都该看看这套共享-私有专家池设计。

论文提出 CoPre-MoE,解决多模态大语言模型持续指令微调中的灾难性遗忘与参数冗余问题。现有 MoE 方法为每个任务单独分配专家,虽避免覆盖共享参数,但导致专家数量随任务线性增长,且 ScienceQA 和 TextVQA 等任务间常出现功能重叠。

Body

CoPre-MoE 的核心机制分三步:先为每个新任务学习任务专属 MoE;再通过功能重叠评估,将跨任务重复的专家合并进固定容量的通用专家池;最后把剩余任务专家分解为共享投影基和紧凑的任务核心,放入私有专家池。

为缓解遗忘,路由器为每个专家配可学习检索键,历史键在持续更新中保持冻结,新增键用正交性惩罚约束。推理时基于输入动态组合两个专家池,无需测试时提供任务标签。实验显示该方法在降低参数开销的同时取得更优性能。

Original · arXiv cs.LG

CoPre-MoE: Consolidating and Preserving MoE for Multimodal Continual Instruction Tuning

Multimodal continual instruction tuning requires multimodal large language models to sequentially acquire new capabilities without catastrophic forgetting. Mixture-of-experts (MoE) architectures offer a promising way to mitigate task interference through expert specialization and combination. However, while existing MoE-based methods avoid overwriting shared parameters by allocating separate experts for each task, this progressive expansion incurs severe cross-task redundancy and linear parameter growth. Specifically, experts across tasks often learn overlapping functions alongside task-specific capabilities. For instance, ScienceQA and TextVQA both locate relevant image regions. To curb parameter redundancy without inducing forgetting, we propose Consolidating and Preserving MoE (CoPre-MoE). For each incoming task, CoPre-MoE first learns task-specific MoE to capture specialized capabilities. To eliminate redundancy and reduce parameter growth across tasks, we design a shared-and-private expert architecture. Specifically, we first evaluate functional overlap to selectively consolidate overlapping experts across tasks into a fixed-capacity generic expert pool. We then factorize the remaining task-specific experts into shared projection bases and compact task-specific cores within a private expert pool. To mitigate forgetting, we construct the router by pairing each expert with learnable retrieval keys, where historical keys remain frozen during continual updates while newly added keys are regularized with an orthogonality penalty. At inference, input-dependent routing dynamically combines both pools without test-time task labels. Extensive experiments demonstrate that CoPre-MoE achieves superior performance while substantially reducing parameter overhead.

Industry Coordinates & Source Verification

This report is categorized under AITOP's "Papers" desk. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include CoPre-MoE, MoE, multimodal, continual learning.

Source attribution: curated from arXiv cs.LG (@Tao Hu, Ling-Xiang Zhang, Da-Wei Zhou).