想研究LLM在社交推理中的真实信念？MafiaScope用探针实验和反事实回放，让你看到智能体表面的伪装。DeepSeek案例数据很硬核。
MafiaScope是一个开源测试平台，将社交推理游戏“黑手党”转化为测量机器理论心智的工具。在32场DeepSeek案例研究中，它收集了13815个结构化探针回答，发现智能体的置信度校准误差为0.17，且高估自己被怀疑的概率达1.5倍。该平台支持交互式可视化、反事实重放，并已发布200+跨模型游戏语料库。
MafiaScope: Non-Invasive, Time-Resolved Belief Probing for LLM Agents in Social Deduction Games
An LLM agent's public behaviour reveals little about its social reasoning: an agent that votes correctly may be guessing, and an agent that lies well leaves no trace of what it actually believes. We present MafiaScope, an open testbed that turns the social deduction game Mafia into a measurement instrument for machine Theory of Mind. After every public utterance, every agent privately answers a configurable set of structured probe questions; the answers never re-enter the game and are scored automatically against the ground truth the engine knows. An interactive visualizer renders the belief trajectories: impersonate mode shows the game as one agent sees it, panels chart timeline-aligned accuracy and calibration, and counterfactual replay forks any recorded step. In a 32-game DeepSeek case study with 13{,}815 parsed probe answers, stated confidence is poorly calibrated, with expected calibration error 0.17, agents over-predict being suspected 1.5 times, and a 30-fork replay experiment walks the counterfactual replay workflow end to end. Engine, viewer and a corpus of 200+ cross-model games are released under an open licence; live demo: https://karpovilia.github.io/mafiascope/; screencast: https://vimeo.com/1208920221.