Tencent paper shows MoE models survive neuron-level pruning
The practical takeaway is that expert granularity is the wrong unit for MoE compression — neuron-level pruning keeps skills intact while cutting memory by over a third. Teams running self-hosted MoE models on constrained GPU budgets, from inference startups to enterprise ML platforms, can apply RAMPr-style scoring to cut serving costs without retraining. The results give open-source MoE deployments a concrete alternative to expert-dropping methods.
A new Tencent research paper finds that domain skills in Mixture of Experts (MoE) models, such as math or coding ability, are stored in scattered neurons across experts rather than confined to individual experts. In their experiments, MoE models retained 91% to 98% of original performance after the least useful 25% of expert neurons were removed.
The researchers introduced RAMP, a measure that scores each neuron's routed contribution across all experts. Neurons selected for one domain overlapped 58% to 74% across samples of that same domain, versus only 15% to 24% across different domains, confirming that skills live in cross-expert neuron groups rather than in dedicated experts. Their pruning method, RAMPr, removes neurons by these scores across all experts instead of dropping whole experts. Tested across 6 models, RAMPr with 50% of neurons remaining reduced GPU memory by 36% to 43% and increased batched prefill throughput by 17% to 39%.
The finding challenges the common assumption that each MoE expert specialises in a domain, since routing mostly tracks token type instead. For teams deploying compressed MoE models, pruning inside experts by routed contribution preserves skills that expert-level pruning would damage.
New Tencent paper shows domain skills (like math or coding ) in Mixture of Experts models sit in scattered neurons across expert.
so cutting weak neurons instead of whole experts shrinks the models while keeping most skills.
In their experiment, Mixture of Experts AI models kept 91% to 98% of their performance after their least useful 25% of expert neurons were cut.
MoE models send each token to a few experts, which suggests each expert handles a domain like math or code. Other work finds routing mostly tracks token type instead.
They scored each neuron's routed contribution with a measure called RAMP. Neurons picked for one domain overlapped 58% to 74% across samples of that domain, versus 15% to 24% across domains, and sat in mixed experts.
RAMPr prunes by these scores across all experts and kept 91% to 98% of original performance with 75% of neurons left, across 6 models. With 50% left, GPU memory fell 36% to 43% and batched prefill throughput rose 17% to 39%.
If you compress MoE models, prune neurons inside experts by their routed contribution instead of treating each expert as a single unit.
This report is categorized under AITOP's "Papers" desk, focusing on China's AI industry and domestic foundation model ecosystem. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include Mixture of Experts, Tencent, model compression, RAMPr.
Source attribution: curated from rohanpaul_ai (@rohanpaul_ai).
