这篇论文提出ROPD，能有效防止微调时被植入有害行为，同时不牺牲专业能力，比现有方法更能抵抗模板攻击。
ROPD通过建模对齐与受损输出概率分布之间的差异，而非拟合特定提示模板，来防御恶意微调攻击。实验在三个数据集（AlpacaEval、SafetyEval、MMLU）和三个基础模型（Llama-2-7B、Mistral-7B、Vicuna-7B）上与四个基线方法（DPO、PPO、GARLIC、SALMON）对比。结果显示基线方法在模板不匹配时下游任务性能严重下降，而ROPD显著缓解模板不匹配风险，防御效果和能力保留均保持强鲁棒性。ROPD虽非完全免疫模板变化，但性能下降远小于现有方法，为LLM安全再对齐设立新标准。
On-Policy Distillation for LLM Safety: A Routing Approach to Template-Robust Realignment
Fine-tuning is the dominant paradigm for specializing large language models (LLMs), yet it exposes a critical vulnerability: malicious data providers can embed harmful behaviors into downstream corpora, creating models that retain professional skills while violating human values on demand. Existing safety-realignment defenses often fail in practice due to three key limitations: they frequently cause catastrophic forgetting of specialized skills; their effectiveness collapses when the defender cannot observe the attacker's prompt template; and successfully realigned models remain susceptible to re-jailbreaking via simple system prompt switches. To address these challenges, we propose Routing-based On-Policy Distillation (ROPD), a novel realignment framework that models the divergence between aligned and compromised output probability distributions rather than fitting specific prompt templates. We conduct extensive experiments comparing ROPD against four state-of-the-art baselines across three datasets and three base models with varying alignment strengths. Our results demonstrate that when baseline defenses face template mismatches, often accompanied by severe degradation in downstream task performance. In contrast, ROPD substantially mitigates template-mismatch risks, maintaining superior robustness in both defense effectiveness and capability preservation. While our analysis indicates ROPD is not entirely immune to template shifts, its performance degradation is negligible compared to existing methods, establishing a new standard for robust LLM realignment.