这篇论文出了个RevengeBench基准,能从行为实验反推LLM的策略代码,测了12个模型恢复率34%-72%,弱模型受益最大。
RevengeBench基准包含75个LLM生成的、基于Elo校准的策略,覆盖5个游戏环境,数据来自CodeClash比赛轨迹。学习者观察隐藏目标策略与对手对战,设计自定义对手策略作为行为探针,并提交可执行假设,用连续动作距离指标评估。在12个前沿LLM上,恢复质量差异显著,从34%到72%的初始距离闭合。重建策略在玩家对玩家锦标赛中提供可衡量的竞争优势,尤其对较弱模型更有帮助。
RevengeBench: Reverse Engineering Code-Space Policies from Behavioral Experiments
For most of scientific history, researchers studying behavior could only infer hidden mechanisms from outward actions: an inverse problem that becomes more tractable when observation is augmented by targeted intervention. We pose a computational analogue: given only behavioral traces of an agent in a game environment, can a learner reconstruct the underlying decision program as executable code, and how much does this reconstruction improve with the ability to design controlled experiments? We introduce RevengeBench, a benchmark of 75 LLM generated, Elo-calibrated policies across five game environments, drawn from CodeClash tournament trajectories. The learner observes the hidden target policy play against sampled opponents and designs behavioral probes in the form of custom opponent policies that elicit informative behavior. It then submits an executable hypothesis, which is evaluated using continuous action-distance metrics. We further validate that recovered code carries informative signal in downstream player-versus-player tournaments. Across twelve frontier LLMs, recovery quality varies substantially (34 to 72% of initial distance closed), with reconstructed policies yielding measurable competitive advantage, particularly for weaker models that otherwise struggle to design effective counter-strategies. Our benchmark positions behavioral recovery of programmatic policies as a tractable inverse problem in code-space, opening a path to opponent modeling, policy interpretability, and the broader question of inferring latent mechanisms from observations.