Epi2Diff框架用LRM推理轨迹预测人类试题难度

Cognitive Episodes in LLM Reasoning Traces Enable Interpretable Human Item Difficulty Prediction

精选理由

想用AI推理过程预测题目难度?这篇论文提出了Epi2Diff,从LRM的思考轨迹中提取片段特征,比直接微调模型效果好8%以上,而且可解释。

AI 摘要

论文提出Epi2Diff(Episode to Difficulty)框架,将大型推理模型(LRM)的推理轨迹映射为认知片段序列,通过推理规模、努力分配和状态转换建模难度。在四个真实人类难度数据集上,Epi2Diff优于微调小语言模型、LLM上下文学习和监督LLM适应等基线。在SAT分类基准上,Epi2Diff相比监督LLM微调获得8.1%平均相对增益。分析发现更难的题目导致更费力、迭代、以实施为中心的片段动态,而非仅更长的回答。

原文 · arXiv cs.AI

Cognitive Episodes in LLM Reasoning Traces Enable Interpretable Human Item Difficulty Prediction

Predicting human item difficulty is central to educational assessment, where reliable estimates support fairness and effective test construction. Existing methods often depend on costly human calibration or item-level textual representations, providing limited evidence about the cognitive processes that make items difficult. We argue that difficulty should be viewed not only as a property of item text, but also as an observable consequence of the problem-solving burden an item induces. Large Reasoning Models (LRMs) offer scalable process evidence through reasoning traces, but such evidence must be structured to support interpretable modeling. To this end, we introduce Epi2Diff (Episode to Difficulty), a framework that maps LRM reasoning traces into cognitively grounded episode sequences. These episodes group trace segments into functional problem-solving states, enabling difficulty to be modeled through reasoning scale, effort allocation, and state transitions. Epi2Diff extracts compact episode-dynamic features and combines them with semantic item representations for human difficulty prediction. Experiments on four real-world human difficulty datasets show that Epi2Diff consistently outperforms strong baselines, including fine-tuned small language models, LLM in-context learning, and supervised LLM adaptation. On SAT-derived classification benchmarks, Epi2Diff achieves an 8.1% average relative gain over supervised LLM fine-tuning baselines. Further analyses show that harder items induce more effortful, iterative, and implementation-centered episode dynamics, rather than merely longer responses. These results demonstrate that cognitive episodes in LRM reasoning traces provide a predictive and interpretable process representation for human item difficulty, offering a new lens for educational measurement with reasoning models.