用 GRPO 训练 Qwen2.5 和 Llama3 学习感知-行动规则,提升 ToM 推理
Learning Perception to Action Rules through Reinforcement Learning for Theory-of-Mind Reasoning
一篇很具体的工作:用 GRPO 让 Qwen2.5 和 Llama3 学 if-then 规则,在四个 ToM 基准上赢过微调和思维链,规则还能给 GPT-4o-mini 用。
论文针对大模型在 Theory-of-Mind(ToM)任务中能推断心理状态却无法用于预测后续行动的问题,生成表达感知与行动关系的 if-then 规则。研究用 GRPO 强化学习方法训练 Qwen2.5 和 Llama3,覆盖四个 ToM 基准。实验显示 GRPO 加规则生成的效果超过 prompting、监督微调和 think reasoning 三种方式。3B 及以上模型生成的规则可迁移给 GPT-4o-mini,提升其下游表现。
Learning Perception to Action Rules through Reinforcement Learning for Theory-of-Mind Reasoning
In Theory-of-Mind (ToM) reasoning, large language models may correctly infer the agent's mental state but fail to use that inference when predicting or judging the agent's subsequent actions. In this work, we investigate whether explicitly connecting perception, belief, and action with rules improves ToM reasoning. We generate concise \textit{if--then} rules that express relations between perception to action and use Group Relative Policy Optimization (GRPO) to train Qwen2.5 and Llama3 models across four ToM benchmarks. GRPO with rule generation outperforms prompting, supervised fine-tuning, and think reasoning. Moreover, rules generated by larger models ($\geq$3B) provide transferable knowledge that improves downstream performance of GPT-4o-mini. These findings suggest that explicit rules help models bridge the reasoning gap in ToM and support knowledge transfer across models.