这篇论文搞了个CHIVE自动找模型意外行为，结果发现现有可解释性技术对预测反事实行为没啥用，但用来训练数据倒挺能泛化，值得做可解释性的人看看。
CHIVE是一个基于智能体的pipeline，通过反事实提示编辑自动发现并研究LLM在自然场景中的意外行为，生成数千条附带反事实证据的高质量解释。研究人员用它评估多种常见可解释性技术，发现这些技术并未提升代理预测反事实行为的能力。此外，用CHIVE生成的数据训练模型预测反事实结果，能够泛化到多种分布外设置。论文表明CHIVE可自动发现自然发生的LLM行为解释，并为评估和改进解释方法提供新途径。
Would this change your answer? Evaluating Explanations of LLM Behavior In The Wild with Counterfactual Experiments
Many areas of AI research, such as language model interpretability and chain of thought faithfulness, seek to explain model behaviors. But what constitutes a "good" explanation? In this work, we evaluate explanations through the lens of counterfactual simulatability-whether the explanation is useful for predicting model behaviors on related counterfactual inputs. To this end, we introduce CHIVE (Counterfactual Hypothesis Investigation Via Edits), a novel agentic pipeline that identifies unexpected model behaviors in the wild and investigates them with counterfactual prompt edits. This yields thousands of high-quality explanations for naturally-occurring model behaviors along with supporting counterfactual evidence. We apply CHIVE in two ways. First, we evaluate whether common LLM interpretability techniques improve an agent's ability to predict counterfactual model behaviors. Surprisingly, we find no uplift from any of the interpretability techniques studied. Second, we use CHIVE to generate training data. We find that training models to predict outcomes of CHIVE-generated counterfactual experiments generalizes to various out-of-distribution settings. Overall, CHIVE automatically discovers explanations of naturally-occurring LLM behaviors, enabling us to evaluate and improve methods for explaining LLM behaviors.