Evaluating Explanation Methods by the Predictors They Induce
This paper provides a new, theoretically grounded method for evaluating how well different model explanation techniques actually work, which is important for anyone using or developing these tools.
A new test for machine learning model explanations is proposed: if an explanation truly describes how a model uses its features, it should be possible to rebuild the model's predictions from it.
The test turns each explanation into a predictor by reading each feature's effect and adding them up, measuring how well that predictor reproduces the model on unseen data.
The test applies to partial dependence plots (PDP), accumulated local effects (ALE), SHAP and LIME, and demonstrates that the best method depends on feature dependence.
Explanations of machine learning models are usually judged by criteria that are hard to compare. We propose a simpler test: if an explanation really describes how a model uses its features, it should be possible to rebuild the model's predictions from it. We turn each explanation into a predictor by reading each feature's effect and adding them up, and measure how well that predictor reproduces the model on unseen data. Nothing is fitted, so the score reflects the explanation itself. The test applies to any explanation that can be written as a function of the features; we demonstrate it on partial dependence plots (PDP), accumulated local effects (ALE), SHAP and LIME. We prove that summing partial dependence curves gives the best possible additive summary of a model when its features are independent, and that this fails when they are dependent. Across 13 real datasets and 9 synthetic designs and four model families, which method scores best depends entirely on feature dependence: where features are independent SHAP is slightly worse than PDP, exactly as the theory predicts; on dependent real data SHAP leads. Some widely used quality metrics even prefer a damaged explanation to an intact one.
This report is categorized under AITOP's "Papers" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include SHAP, LIME.
Source attribution: curated from arXiv cs.LG (@Jacob Selbæk, Hugo L. Hammer).