ModRoute：按样本路由模态提升 VLM 遗忘效果
Words Before Pixels: Selective Modality Routing for Vision-Language Model Unlearning
一篇研究 VLM 遗忘的论文，发现用文本来表示要删掉的坏行为比用图像更管用，还给出按样本自动选模态的 ModRoute 方法，分数提升挺明显。
论文研究视觉语言模型（VLM）的遗忘问题，认为遗忘数据在优化前如何表示，与损失目标设计同等重要。作者将 VLGuard 等基准拆解为纯图像、纯文本和多模态三种版本做对照实验，发现不安全指令跟随行为用文本表示抑制效果更好，但纯文本遗忘会削弱依赖视觉证据的视觉对齐能力。基于这一权衡，论文提出梯度引导的按样本模态路由方法 ModRoute，把视觉压力高的样本转为纯文本、其余保留多模态。在 LLaVA-1.5-7B 上的 VLGuard 测试中，ModRoute 在 γ=0.5 时取得 0.847 的综合遗忘-效用分数，比最佳随机切换高 2.5%，比纯多模态或纯文本遗忘高出至少 46%。
Words Before Pixels: Selective Modality Routing for Vision-Language Model Unlearning
Vision-language model (VLM) unlearning is often treated as an objective-design problem: given multimodal forget data, the goal is to optimize a loss that removes undesirable behavior while preserving retained capabilities. We argue that this view overlooks an equally important data-centric question: how should each forget example be represented before optimization? Existing methods typically keep image-conditioned failures in their original image-text form, implicitly assuming that pixels are the right route for unlearning whenever pixels appear in the input. We challenge this assumption by constructing modality-decomposed versions of VLM benchmarks, enabling controlled comparisons among image-only, text-only, and multimodal representations of the same examples. Our modality-mixing experiments show that unsafe instruction-following behavior is more effectively suppressed through textual representations than through images alone, while purely text-only unlearning can still weaken visual grounding when the target behavior depends on visual evidence. Motivated by this trade-off, we introduce ModRoute, a gradient-guided per-sample modality routing strategy that selectively converts high visual-pressure examples to text-only form while keeping the remaining examples multimodal. Notably, on VLGuard with LLaVA-1.5-7B, ModRoute has the strongest composite unlearning-utility score of 0.847 at $γ=0.5$, a 2.5% improvement over the best random-switching score and $\geq$46% better than pure multimodal or text unlearning. Overall, our results show that effective VLM unlearning depends not only on the objective, but also on the composition and per-sample modality routing of the unlearning data.