做多模态持续学习的团队终于有了解决任务路由混乱的方案——ProtoAda通过原型感知输出结构，避免VQA和接地任务互相污染，建议关注论文中的几何整合细节。
多模态大语言模型（MLLMs）通过指令微调表现出色，但实际部署需要持续获取新的视觉语言能力，因此多模态持续指令微调（MCIT）至关重要。现有方法常采用稀疏架构（如混合LoRA专家）通过图像-文本相似度路由，但任务响应结构不同时可能共享高度相似的语义，导致路由错误和梯度干扰。ProtoAda提出格式感知的任务原型，将任务分配与路由对齐到语义和输出结构，并通过几何感知方式整合格式兼容的更新，有效重用和优化现有参数。实验表明，ProtoAda在多个基准上表现优异，尤其对答案结构易被顺序微调破坏的任务效果显著。
ProtoAda: Prototype-Guided Adaptive Adapter Expansion and Geometric Consolidation for Multimodal Continual Instruction Tuning
Multimodal Large Language Models (MLLMs) achieve strong performance through instruction tuning, but real-world deployment requires them to continually acquire new vision-language capabilities, making Multimodal Continual Instruction Tuning (MCIT) essential. To reduce inter-task interference and promote collaboration, recent methods often employ sparse architectures like Mixture of LoRA Experts with image-text similarity routing. However, tasks with distinct response structures could share highly similar visual-linguistic semantics and thus be wrongly routed to the same expert; image-text similarity alone is insufficient for reliable task assignment. For example, an expert in a grounding task requiring coordinate prediction may be biased toward producing short textual answers after learning semantically similar VQA tasks. This format-blind task assignment integrates heterogeneous response types into shared parameters, inducing gradient interference and ineffective expert collaboration. To address this problem, we propose ProtoAda, a prototype-guided adaptive tuning framework. ProtoAda introduces format-aware task prototypes to align task assignment and routing with both task semantics and output structure, and further consolidates format-compatible updates in a geometry-aware manner to effectively reuse and progressively refine existing parameters. Extensive experiments on multiple benchmarks demonstrate that ProtoAda achieves superior performance, especially on tasks whose answer structures are easily corrupted by sequential tuning.