论文:在非完全信息博弈中逐节点评估语言模型智能体决策
Instrumenting Failure: Scoring Language Model Agents at Every Decision Node in Imperfect-Information Games
传统智能体评估只看任务成败,这篇论文把五个非完全信息博弈拆到每个决策节点逐项打分,还用 CFR+ 均衡当参照。做智能体评测或研究模型推理行为的人能直接借鉴这套方法和 gate 机制。
arXiv 论文 2611.20644 提出一种智能体评估方法:对三个 12B 到 14B 的指令微调语言模型在五种零和非完全信息博弈中的每个决策节点逐项打分,评估维度来自决策论与博弈论的八项性质,其中五项以 CFR+ 均衡为参照。模型在每个节点还需通过单独调用陈述对对手隐藏状态的信念,无记忆且温度为 0 时,陈述信念与均衡后验的总变差距离为 0.250 到 0.288,动作与信念的一致性超出随机水平 -0.04 到 0.19。
评估还发现:在单一提示与贪心解码下,38 个参照策略会随状态改变动作的格子中有 10 个里模型动作完全不变;引入记忆后,13 个可测试阶梯中有 9 个降低了恒定动作比例;温度 0.2 时恒定动作的格子从 46 个中的 16 个降到 5 个,而信念维度结果保持稳定。
方法上带有一个 gate 机制,对运行无法支持的信念分数给出理由并拒绝打分,八个非零缺陷率中七个可归因于测量工具、某种表示或模型本身。相比只报告任务成功率的常规智能体评估,这种逐节点打分能定位模型在哪个具体决策上出错。
Instrumenting Failure: Scoring Language Model Agents at Every Decision Node in Imperfect-Information Games
Agent evaluations report task success and rarely say what an agent did at each decision. We score every decision node of three 12B to 14B instruct-tuned language models (LMs) playing five zero-sum imperfect-information games against four opponents on eight properties from decision and game theory, five of them against a CFR+ equilibrium reference. At each node, the LM also states a belief about the opponent's hidden state in a separate call. Without memory and at temperature 0, stated beliefs sit 0.250 to 0.288 in total variation distance from the equilibrium posterior, and where the action responds to the state its agreement with that belief exceeds chance by -0.04 to 0.19. The two calls frame the opponent differently, so the result is a behavioural inconsistency and makes no claim about internal beliefs. Under one prompt and greedy decoding, the LM's action never varies with the state in 10 of the 38 cells where the reference varies its own. Memory lowers the constant-action share in 9 of 13 testable ladders at temperature 0; at temperature 0.2 the cells at 1 fall from 16 to 5 of the 46 informative cells, while the belief results hold. A gate withholds, with a stated reason, any belief score the run cannot support, and seven of the eight non-zero defect rates are attributable to the instrument, a representation, or the LMs.
This report is categorized under AITOP's "Papers" desk. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include LLM, agents, game theory, CFR+.
Source attribution: curated from arXiv cs.AI (@Michael Kingston, Sonit Singh, Alan Blair).