这篇论文提出了WAG方法，能精准定位LLM中那些一旦改动就会崩掉的关键参数，比传统重要性指标更灵敏，还顺手解决了专家分配、遗忘、量化等实际问题。
论文提出Weight-Adjusted Gradients方法，通过捕获权重与一阶梯度的交互来估计参数重要性。在GPT-2、Llama-2等模型上，WAG发现仅占0.3%的关键参数，修改它们会导致性能下降超过40%，而现有幅度剪枝准则遗漏了这一失效模式。该方法在混合专家模型分配、参数级遗忘、混合精度量化和知识编辑层选择等四类应用中展示了有效性。实验表明WAG统一了零阶与一阶信息，为分析、调试和控制LLM提供了新工具。
Weight-Adjusted Gradients Reveal Parameter Importance and Failure Modes in LLMs
Understanding which parameters are influential in Large Language Models (LLMs) is central to improving their efficiency, reliability, and interpretability. We introduce Weight-Adjusted Gradients (WAG), a simple yet effective approach for estimating parameter importance that explicitly captures the interaction between model weights and first-order gradient information and identifies parameters that disproportionately influence model behavior, such as those responsible for collapse phenomena in LLMs. Across a range of models and settings, we show that WAG surfaces a tiny but critical subset of parameters whose modification leads to dramatic degradation in performance, a failure mode that existing importance metrics overlook. These findings reveal a previously underexplored interplay between weights and gradients, suggesting that parameter importance cannot be fully understood through either signal alone. The surprising effectiveness of WAG points to fundamental structural properties of trained networks and motivates new open questions about the role of zeroth-order and first-order information in deep learning. We demonstrate the practical utility of WAG across multiple applications, including expert allocation in mixture-of-expert architectures, parameter-specific unlearning, mixed-precision quantization, and layer selection for knowledge editing. Our results position WAG as a unified approach for analyzing, debugging, and controlling LLMs, and opens new directions for principled model-level interpretation.