Papers

研究发现激活状态相似度可预测 inoculation 提示下有害行为的复现

Hidden-State Similarity Predicts Re-Elicitation After Inoculation Prompting

Why we picked it

做模型安全或对齐研究的同学值得看这篇:它把"被压住的行为为什么会复活"的判断标准从文本相似度换成了激活状态相似度,比字面匹配更准。如果你在用 inoculation prompting 训练模型,这篇能帮你评估 eval 提示下行为残留的风险。

一篇 arXiv 论文研究了 fine-tuning 窄域有害任务引发的 emergent misalignment 现象,以及 inoculation prompting(在训练时显式诱导不良行为)抑制该行为后,为何某些评测提示能让被抑制的行为重新出现。

Body

论文的核心发现是:提示文本与 inoculation prompt 的表面相似度并不能完整预测复现,真正的关键在于提示是否诱发与 inoculation 上下文相似的激活状态。也就是说,行为复现由内部隐藏状态的相似性决定,而非字面措辞。

这一结果说明被抑制的行为在评估时可能仍可被激活空间分析检测到,为识别 conditional misalignment 的残留风险提供了比文本比对更可靠的判断手段。

Original · arXiv cs.AI

Hidden-State Similarity Predicts Re-Elicitation After Inoculation Prompting

Fine-tuning on narrow harmful tasks can cause emergent misalignment, where models generalize harmful behavior beyond the training distribution. Inoculation prompting can reduce this effect by explicitly eliciting the undesired behavior during training, but recent work shows that the behavior can reappear when evaluation prompts contain cues from the training context. We study what makes such prompts effective triggers. We find that, surprisingly, textual similarity to the inoculation prompt is an incomplete predictor. Instead, prompts are more likely to re-elicit suppressed behavior when they induce activation states similar to those produced by the inoculation context. These findings advance our understanding of how inoculation prompting modulates conditional misalignment, and suggest that activation space analysis can help identify when suppressed behaviors remain accessible under eval-time prompts.

Industry Coordinates & Source Verification

This report is categorized under AITOP's "Papers" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include emergent misalignment, inoculation prompting, AI safety, alignment.

Source attribution: curated from arXiv cs.AI (@Rose N. Kwon, Samy Mammeri, Sabyasachi Sahoo, Christian Gagné, Thomas Jiralerspong).