YOPO:冻结语言模型单次前向传播同时作答与弃权

You Only Pass Once: Answering and Abstaining Together in a Single Forward Pass of a Frozen Language Model

精选理由

YOPO 让冻结的 Qwen2.5 单次前向传播就完成作答和弃权判断,alphaNLI 达 0.798,超过两遍推理,值得看。

AI 摘要

YOPO 在冻结的 Qwen2.5(1.5B/3B/7B)上实现单次前向传播同时作答和弃权。它用条件转向探针改写残差流,并用零样本充足性方向判断信息是否足够,同时训练小网络从转向后的状态重建转向前的残差。在 alphaNLI 上,1.5B 模型的三路准确率从 0.375 提升到 0.798,超过两遍推理参考的 0.753。在十个骨干模型、六个模型家族上,YOPO 在 1.5B/3B/7B 分别取得 0.798/0.830/0.893,均高于两遍参考。论文在 SQuAD2、RepLiQA、MuSiQue 上完成原生标签验证,并提出首个 answer-or-abstain 基准。

原文 · arXiv cs.LG

You Only Pass Once: Answering and Abstaining Together in a Single Forward Pass of a Frozen Language Model

A frozen language model on reasoning tasks has two coupled weaknesses: it under-uses evidence its own residual stream already encodes, and it fails to detect when the input is insufficient to answer, so it confabulates. This paper consolidates two research lines that address these on the same residual stream: a conditional steering probe writes the stream at mid-stack layers and recovers reasoning accuracy from a frozen backbone, and a zero-shot sufficiency direction reads the stream and abstains when information is insufficient. Deployed in one forward pass they interfere: the steering write shifts the state the direction reads, costing up to 8 AUROC points of cross-domain transfer on small models; a separate clean pass doubles inference cost. We keep the direction fixed and train a small network to reconstruct the pre-steering residual from the steered one -- mean-squared error on (steered, clean) pairs, no sufficiency labels -- and read the direction on the reconstruction. The resulting system, YOPO (You Only Pass Once), answers, steers, and abstains in one forward pass of a frozen Qwen2.5 backbone (1.5B/3B/7B). End to end, three-way accuracy more than doubles the frozen baseline (0.375->0.798 on 1.5B alphaNLI) and one pass beats the two-pass reference at every scale (0.798/0.830/0.893 vs 0.753/0.790/0.863) and on ten backbones across six model families. We chart the capacity-transfer frontier quantifying the principle that abstention should not be trained in; a source-side audit catches our own alphaNLI construction leaking a surface artifact, so architectural claims are anchored on native-label replications (SQuAD2, RepLiQA, MuSiQue); and on the standard four-domain suite we contribute, to our knowledge, the first answer-or-abstain benchmark, where our gate tops every in-domain dataset and the label-free direction is the only gate family to survive domain transfer.