可验证的社会推理框架 Fuse 用于 LLM 助手研究
Verifiable Social Reasoning for LLM Assistants
朋友研究 LLM 社交推理的，这篇论文开源了 Fuse 框架和 21k 数据集，能帮你系统分析 LLM 在社交场景下的表现。
论文提出 Fuse 框架，通过多代理模拟研究 LLM 在社交咨询场景下的推理能力。该框架让一个有隐藏动机的目标代理与用户代理互动，用户再咨询 LLM 推断目标动机，以此构建可验证的基准。通过 24k 人类标注验证模拟真实性，并测试了 12 个 LLM，发现用户干预会加剧推理难度，模型对用户偏见敏感，且更长的对话不一定提升性能。
Verifiable Social Reasoning for LLM Assistants
LLM assistants are widely used for daily social advice, yet evaluating their social reasoning in such consultation settings remains challenging since (i) it requires setups where the assistant learns about social situations from subjective user narratives, and (ii) social properties, such as others' intentions, typically lack verifiable ground truth. To address these challenges, we introduce Fuse, a multi-agent simulation framework for studying user-mediated social reasoning. In Fuse, a target agent with a hidden motive interacts with other agents including one representing the user, who then consults the evaluated assistant to infer the target's motive, providing verifiable ground truth by construction. Simulation faithfulness is validated through a human study with 24k annotations. We apply Fuse to 12 LLMs and demonstrate its analytical utility by systematically isolating key factors, showing that (i) user mediation compounds the inherent difficulty of social reasoning; (ii) LLMs exhibit systematic sensitivity to biased user framing; (iii) models can require more details than humans need to reach a correct prediction; and (iv) longer conversations do not always improve performance despite providing opportunities for clarifying questions. We open-source Fuse and a dataset with 21k examples.