这项研究解决了推理模型行为控制中输出质量下降的痛点,做模型对齐或安全控制的开发者可以直接用FPCG方法,在保持生成质量的同时精准引导模型行为。
大型推理模型(LRM)在部署时常出现意外行为,现有测试时干预方法(如激活引导)虽能控制输出,但会降低输出质量。研究发现,传统方法依赖的“检测特征”只能识别已生成文本中的行为,对未来行为预测能力差。研究者训练了激活探针,能从中间推理步骤预测未来行为概率,准确率达64%-91%。基于此,他们提出文本级引导方法FPCG(未来探针控制生成),通过采样多个候选句子并选择未来行为概率最高的一个,实现几乎无质量损失的引导。实验表明,FPCG在多个激活引导失效的场景中仍有效,为控制LRM行为提供了更精细的途径。
Predicting Future Behaviors in Reasoning Models Enables Better Steering
Deployed large reasoning models (LRMs) often behave unexpectedly. Test-time steering controls LRM outputs by intervening on their hidden representations, but it can degrade output quality. We argue that prior steering work implicitly relies on internal features that detect behavior in already generated text. We show that these detection features are poor predictors of future behavioral outcomes, and thus not the natural intervention target. Instead, we train activation probes to predict future behavior likelihoods from intermediate reasoning steps. These probes predict the most likely behavior with 64%-91% accuracy, revealing a separate type of internal prediction features. Building on these prediction features, we introduce a text-level steering method, Future Probe Controlled Generation. FPCG samples multiple candidate sentences and chooses the best one according to a probe predicting the future behavior likelihood. This enables steering with almost no output quality degradation. FPCG also enables steering in several evaluations where activation steering fails. These results show that distinguishing detection and prediction features enables a more nuanced approach to controlling LRM behaviors.