ThoughtFold解决了LRM过度思考的痛点，做推理模型优化的团队可以直接参考其内省偏好学习方法，能大幅降低计算成本而不牺牲精度。
大型推理模型（LRM）在链式思维（CoT）上通过可验证奖励强化学习（RLVR）取得了显著进展，但长CoT中固有的试错和冗余探索被强化，导致过度思考问题。现有方法主要偏向较短轨迹，但学习信号仍基于结果，无法减少长CoT中的冗余记忆。为此，研究者提出ThoughtFold框架，通过细粒度偏好学习来缓解冗余探索，实现高效推理。它采用内省策略识别正确轨迹中的冗余，生成候选子轨迹谱，并引入掩码偏好优化目标，显式惩罚冗余探索，鼓励模型直接连接关键推理段，从而折叠推理链。实验表明，ThoughtFold将DeepSeek-R1-Distill-Qwen-7B的token使用量减少约56%，同时保持最先进的准确性。
ThoughtFold: Folding Reasoning Chains via Introspective Preference Learning
Large Reasoning Models (LRMs) have achieved remarkable progress thanks to Reinforcement Learning with Verifiable Rewards (RLVR) on Chain-of-Thoughts (CoTs). However, since long CoTs naturally contain trial and errors and mainstream RLVR approaches choose outcome-correct CoT trajectories for memorization, the redundant explorations in long CoTs are inevitably reinforced, which results in the over-thinking issues of LRMs. Previous attempts to resolve this issue mainly give more advantage to shorter trajectories, yet their learning signals are still outcome-based and cannot reduce the memorization of redundant explorations in long CoTs. Therefore, we propose ThoughtFold, a framework that leverages fine-grained preference learning to mitigate redundant explorations for efficient reasoning. ThoughtFold employs an introspective strategy to identify redundancy within each correct trajectory, which yields a spectrum of candidate sub-trajectories. Leveraging this spectrum, we introduce a masked preference optimization objective that explicitly penalizes redundant explorations and encourages the model to directly bridge essential reasoning segments, effectively folding its reasoning chains into a more concise path. Extensive experiments show that ThoughtFold significantly enhances efficiency. It reduces the token usage of DeepSeek-R1-Distill-Qwen-7B by approximately 56% while maintaining state-of-the-art accuracy.