用条件分布检验定位智能体与玩家行为的分歧
When Agents Play Differently: Localizing Human--Agent Behavioral Divergence with Conditional Distributional Testing
搞游戏 bot 测试或 agent 评测的可以看看这篇:它不比平均分,而是按上下文分组定位 bot 在哪类模式里玩得不像人,代码也开源了。比单看胜率或聚合分数的常规评测精细得多,做自动化 playtesting 的团队用得上。
一篇 arXiv 论文提出分层检验框架,在上下文分格内计算非参数差异统计量,并自动发现智能体行为偏离玩家的子群规则。
核心发现是:会话级指标以输赢结果为主,其信号量不超过均值比较;而走子效率等过程级指标才能揭示真实的行为差异。
作者将该方法应用于一款商业三消游戏的 playtesting bot,结果显示该 bot 在部分游戏模式下与人类玩家存在显著分歧,在另一些模式下则未检测到分歧,这类模式级的差异在聚合指标下不可见。实现代码已开源,便于复现与落地。
When Agents Play Differently: Localizing Human--Agent Behavioral Divergence with Conditional Distributional Testing
Evaluating whether artificial agents behave like humans requires comparing not just average outcomes but entire behavioral distributions across diverse contexts. We propose a stratified framework that computes nonparametric discrepancy statistics within contextual cells, and discovers subgroup rules identifying where agent behavior departs from human play. We show that session-level metrics dominated by win/loss outcomes offer little signal beyond mean comparisons, whereas process-level metrics such as move efficiency reveal genuine behavioral differences. Applied to a commercial match-3 game, it uncovers patterns invisible to aggregate metrics: a playtesting bot is diverging substantially from humans in certain game modes, while showing no detectable divergence in others. We release our implementation as open source to support reproducibility and adoption.
This report is categorized under AITOP's "Papers" desk. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include agents, agent evaluation, open source.
Source attribution: curated from arXiv cs.AI (@Oleg Smirnov, John Pertoft, Sara Karimi).