这篇论文点出一个关键漏洞：模型准、解释对，不等于模型真懂现象。做AI科学发现的人必看。
本文指出，后验解释方法常用于解释科学机器学习模型，但仅凭模型可靠（预测匹配结果）和解释忠实（解释匹配模型）不足以证明模型揭示了现象的真实结构。文章强调，两者均未验证模型的工作机制是否与现象本身一致。作者认为，要支持关于现象结构的声明，还需依赖外部佐证，而非仅靠解释链。
Reliability, Faithfulness, and the Limits of Post-hoc Explanations of Opaque Scientific Models
Post-hoc explanation methods are routinely used to interpret scientific machine learning models, with the deliverable understood to be insight into the phenomenon the model has been trained on. The transition may be taken to be secured once the model is reliable enough and the explanation faithful enough. We argue it is not. Reliability checks that the model's predictions match the phenomenon's outcomes, and faithfulness checks that the explanation matches the model, but neither checks whether the model works as the phenomenon works, which is what a claim about structure requires. The chain can support candidate hypotheses under external corroboration, but it cannot, on its own, support claims about how the phenomenon is in fact structured.