Papers

GEAR²:让被剪枝的 MoE 专家通过门控复用继续发挥作用

GEAR$^2$: Pruned Experts Live On through Gated Reuse of Activated Experts and a Missing-Mass Residual

Why we picked it

剪 MoE 专家不用再花钱做恢复训练了。GEAR 直接复用已激活专家的输出,GEAR² 加个轻量残差补缺口,在 16B 到 106B 的五个模型上验证有效,剪一半专家性能还能保住。做模型部署和显存压缩的工程团队可以试试。

一篇 arXiv 论文提出 GEAR 和 GEAR² 两种免训练方法,用于在专家剪枝后恢复被丢弃的 MoE 专家贡献。

Body

作者发现被剪枝专家的功能可以从剩余激活的专家中重建,GEAR 通过门控直接复用可靠激活专家的缩放输出,无需训练或额外专家计算。GEAR² 在此基础上加入一个超轻量残差,补偿复用无法覆盖的部分。

在 16B 到 106B 参数的五个 MoE LLM 上剪枝 25% 和 50% 专家时,GEAR 在九个零样本基准上准确率比免训练基线最高提升 4 个百分点,GEAR² 在所有设置下优于基线方法最高 2.75 个百分点,剪枝越激进(最高 50%)优势越明显(最高 2.

72 个百分点)。相比静态替代或专家合并方案,该方法不改变专家原有功能,也省去了昂贵的恢复训练,对需要压缩 MoE 模型显存占用的部署方有直接参考价值。

Original · arXiv cs.AI

GEAR$^2$: Pruned Experts Live On through Gated Reuse of Activated Experts and a Missing-Mass Residual

Mixture-of-Experts (MoE) Large Language Models (LLMs) activate only a few experts per token while generally keeping all the experts in GPU memory for high efficiency. Expert pruning reduces this GPU memory footprint cost, yet discards the contribution of pruned experts. Existing remedies either rely on static substitutes, merge experts at the cost of altering their functions, or require costly recovery training. In this paper, we find that the functionalities of pruned experts can be efficiently reconstructed from the activated remaining experts. We propose GEAR (Gated Expert Activation Reuse), which reuses the scaled outputs of reliable activated experts without any training or additional expert computation. In addition, we propose GEAR$^2$ to further recover what reuse cannot express with an ultra-lightweight residual. On five MoE LLMs from 16B to 106B parameters with 25% and 50% of experts pruned, GEAR improves over its training-free counterparts in nearly every setting (up to 4 pp.) in terms of accuracy, and GEAR$^2$ outperforms baseline approaches (up to 2.75 pp.) in every setting on nine zero-shot benchmarks, with larger advantages (up to 2.72 pp. higher) under more aggressive pruning (up to 50%).

Industry Coordinates & Source Verification

This report is categorized under AITOP's "Papers" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include MoE, model pruning, LLM inference.

Source attribution: curated from arXiv cs.AI (@Linliang Chen, Yan Zhong, Wang Kang, Sai Li, Ji Wu, Ji Liu).