失败推理轨迹揭示可修复性:无需阅读文本,仅凭分布特征即可诊断

Failed Reasoning Traces Tell You What Is Fixable (But Not by Reading Them)

精选理由

这篇论文为AI推理失败提供了诊断工具,做模型调试和推理优化的团队可以直接用这三个特征来区分可修复与不可修复的失败,无需额外训练或权重访问,值得关注。

AI 摘要

该论文提出,大语言模型在推理任务中失败时,常见的做法是增加采样次数重试,但忽略了失败轨迹本身蕴含的宝贵信息。作者发现,失败可分为“运气不好”和“结构性问题”两类,前者可通过更多采样解决,后者则无法通过重试修复。他们从失败轨迹的分布特征中提取了三个问题级特征,这些特征不依赖轨迹文本,而是基于可用的测试时干预结构。这些特征能以84.3%的准确率聚类失败模式,并支持一种无需训练的路径选择规则,在Steerable-Hard子集上将修复成功率提升12.2%。该方法无需访问训练过程或模型权重,即可用于测试时路由和后训练分析。

原文 · arXiv cs.AI

Failed Reasoning Traces Tell You What Is Fixable (But Not by Reading Them)

When post-trained language models fail on reasoning problems, the common test-time-scaling response is to spend more compute on additional attempts, and the failed traces play no further role. We argue this discards a crucial signal; some failures come from unlucky sampling, where more rollouts help, while others are structural and resist resampling regardless of budget. We propose that failed traces encode recoverability structure: the inference-time signature of which test-time interventions can rescue a given failure. Three problem-level trajectory features, derived from the structure of available interventions, recover this structure from the distributional signature of failed rollouts, not their text. They cluster failures into stable regimes, characterize the failure topography of different post-training methods ($84.3{\pm}4.3\%$ accuracy, $+20\%$ over a majority-class baseline), and support a training-free routing rule that lifts rescue by $+12.2\%$ on the deployment-relevant Steerable-Hard subset (failures where retry is insufficient and a bounded intervention is reachable). The features and the routing rule transfer across two cross-family probes. The same three features thus convert failed traces from discarded data into a diagnostic object, supporting test-time routing and post-training analysis without training-time or weight-space access.

失败推理轨迹揭示可修复性:无需阅读文本,仅凭分布特征即可诊断 · AI 热点