AITOPAI热报
PapersSelected73°

Mixture-of-Experts Inference Optimization Study

Launch-Bound and Substitutable: Why Three Inference Optimizations Fail to Pay Off in Mixture-of-Experts Models

arXiv: DeepSeek@Gokulakannan Sakthivel et al. (4)
Paper
Read the original
Why we picked it

The findings impact MoE deployment efficiency for researchers and companies using these models in production environments.

The paper examines three inference optimizations on OLMoE-1B-7B, DeepSeek-V2-Lite, and Qwen3-30B-A3B models. Fused Triton kernels show 5.6x to 9.0x improvement in isolation but only 0.999x end-to-end against a 1.07x ceiling. INT4 quantization changes 0.53 of eight experts per token, yet reproduces only 2.7% of quality loss, indicating substitutability. Removing 23 torch.compile graph breaks makes the model three times slower.

Original · arXiv: DeepSeek

Launch-Bound and Substitutable: Why Three Inference Optimizations Fail to Pay Off in Mixture-of-Experts Models

Mixture-of-Experts (MoE) models route each token to a few of many expert networks, and that routing is data- dependent in a way standard inference optimizations do not ex- pect. This paper measures what three of them actually deliver on OLMoE-1B-7B, DeepSeek-V2-Lite, and Qwen3-30B-A3B. Fused Triton kernels reach 5.6x to 9.0x in isolation but 0.999x end to end against a measured 1.07x ceiling, because the model spends its time waiting on roughly a thousand kernel launches per forward pass rather than on the arithmetic those kernels improve. INT4 quantization changes on average 0.53 of the eight selected experts per token position, yet replaying exactly those changed routes through full-precision weights reproduces only 2.7% of the quality loss, which makes the experts substitutable rather than specialized. Removing all 23 torch.compile graph breaks, the step prior work treats as the structural fix, makes the model three times slower. A fourth result ties the three together: leaving the routers in FP16 lowers drift by 20% while raising loss, so routing fidelity and output quality are separable objectives. Every number recomputes from committed per-token route dumps.

#Mixture-of-Experts#OLMoE-1B-7B#DeepSeek-V2-Lite#Qwen3-30B-A3B#inference optimization

Keep reading

Recent stories with the same tag or section
How-toBuildersMulti-sourceSelected00:03
PyTorchCon North America 2026: five NVIDIA sessions to watch

The session list doubles as a map of where PyTorch engineering effort is heading: communication-computation overlap for MoE training and dynamic-shaped CUDA graphs signal that large-model training performance is the priority theme. Engineers building or tuning training pipelines can use these talks to pick up concrete scaling techniques, including the 10x data loader speedup. The Ecosystem Landscape talk is also useful for open-source maintainers seeking official PyTorch ecosystem status.

#PyTorch#PyTorchCon 2026#NVIDIA#CUDA
Event
官方账号PyTorch@PyTorch11 sourcesSource
How-toBuildersSelected02:27
Simon Willison asks which open-weight MoE model suits coding under 60GB

The 60GB RAM ceiling and 12 tokens/second floor describe the practical envelope many local-model enthusiasts work inside, and the replies surface which open MoE checkpoints people actually run for coding on that budget. Practitioners choosing hardware or deciding between dense and MoE models for local coding assistants can mine the thread for tested configurations. It also reflects how community recommendations now rival benchmarks in model selection.

#open-source models#Mixture-of-Experts#coding assistants#Simon Willison
官方账号Simon Willison@simonwSource
PapersBuilders12:11
Chinese content提出无限参数LLM架构，可实时生成权重适应新数据

这个研究挺有意思的，作者想解决传统LLM在运行时无法学习新知识的问题，提出的无限参数架构挺有创意，通过实时生成权重来适应新数据，比单纯用提示词或检索的方法可能更高效。

#Infinite-Parameter LLM#Mixture-of-Experts#提示词工程
arXiv cs.LG@Jinli Hu et al. (4)Source
PapersBuildersSelected11:53
Mixtures-of-Experts models overfit more to repeated data

This matters to developers working with large language models because it highlights a critical trade-off: MoE models, while more efficient, are more vulnerable to overfitting when training data is repeated, potentially leading to worse performance in real-world applications.

#Mixture-of-Experts#data repetition#LLM#model sparsity
arXiv cs.LG@Atindra Jha et al. (5)Source