MoE 专家重要性因果审计：观测指标无法预测剪枝效果
From Observation to Intervention: A Causal Audit of Expert Importance in Mixture-of-Experts Models
MoE 模型剪枝的常用假设被实验证伪了——做模型压缩或可解释性研究的团队，建议重新审视你的专家选择策略，别再依赖路由统计指标。
这篇论文对混合专家模型（MoE）中的专家重要性评估方法进行了因果审计。研究者发现，当前广泛使用的路由统计指标（如利用率、激活范数、路由权重分布）无法预测哪些专家可以被移除而不影响模型功能。
他们在 OLMoE-1B-7B-0924、Qwen1.5-MoE-A2.
7B 和 DeepSeek-V2-Lite 三个高冗余 MoE 架构上进行了 token 级干预实验，结果在所有 60 个指标-层组合中，观测指标与因果重要性之间的效应量均低于 Cohen's d = 0.17。
现有剪枝方法之所以有效，并非因为它们识别出了可去除的专家，而是因为早期层的冗余性使得大多数选择标准可以互换。这项研究为可解释性领域提供了一个明确的反例，说明从总体观测统计到 token 级干预结论的推理步骤需要更严格的因果验证。
From Observation to Intervention: A Causal Audit of Expert Importance in Mixture-of-Experts Models
Interpretability methods routinely use population-level summary statistics over observed model behaviour to license claims about the effects of targeted interventions on specific computations; in Pearl's terms, they treat rung-1 associational evidence as if it supported rung-2 interventional conclusions, a move whose validity is rarely tested. We examine one concrete instance: the use of routing statistics in Mixture-of-Experts (MoE) pruning, where utilization rates, activation norms, and routing weight distributions are treated as predictors of which experts can be removed without functional cost. A token-level interventional audit across three high-redundancy MoE architectures (OLMoE-1B-7B-0924, Qwen1.5-MoE-A2.7B, DeepSeek-V2-Lite) finds no observational metric predicts causal expert importance after multiple-comparison correction in any model, with effect sizes below Cohen's $d = 0.17$ across all 60 metric-layer combinations. A per-token routing weight control rules out insufficient power, recovering a single Bonferroni-significant signal at OLMoE's final MoE layer ($d = +0.231$, $p = 0.0013$). Existing pruning methods succeed in this regime not by identifying dispensable experts but because early-layer redundancy renders most selection criteria interchangeable. Our results provide an explicit counterexample to the common inferential step from population-level observational summaries to token-level interventional claims about expert importance, and illustrate how interventional audits can calibrate the evidential standards for interpretability claims.
This report is categorized under AITOP's "Papers" desk. Key entities and technical topics under continuous tracking include MoE/混合专家, 模型剪枝, 因果推断, 可解释性.
Primary source verification: published directly by arXiv: DeepSeek with first-party authority.
