这篇论文用Elenchos框架测了LLM的溯因推理，发现模型能发现异常但猜不准具体改了啥，挺有意思的。
Elenchos是一个基于lambda演算系统的生成式评估框架，用于测试大语言模型（LLM）的溯因推理能力。评估发现，前沿和中端模型在检测系统是否被修改时表现较好，但识别具体规则修改的能力显著不足。在交互突变场景下，模型性能大幅下降，通常只能恢复部分突变。初步证据显示，增加推理时间预算仅带来微小改进，收益递减。
LLMs Can See the Smoke but not the Fire: Evaluating Abductive Reasoning with Elenchos
Large language models (LLMs) excel at pattern recognition and text generation, but their capacity for abductive inference - inferring latent hypotheses that explain observed behavior - remains poorly understood. Here, we introduce Elenchos (named after the Socratic method of cross-examination), a generative evaluation framework that measures abductive reasoning as a structural inverse problem. Given a reference formal system, such as the lambda-calculus, and a potentially mutated counterpart, agents must determine whether a mutation has occurred and infer the rule modifications responsible for the resulting behavioral differences. Evaluating frontier and mid-tier LLMs reveals a consistent detection-attribution dissociation: models often recognize that a system has been altered but struggle to identify the latent mutations causing the observed discrepancies. Performance degrades substantially under interacting mutations, where models frequently recover only a subset of the underlying mutations. Preliminary evidence also suggests diminishing returns from increased inference-time reasoning, with only modest improvements under larger reasoning budgets, though this finding requires further validation.