MobileMoE解决了移动端大模型部署的算力与内存瓶颈，做端侧AI应用或手机端推理的开发者可以直接参考其架构和训练方案，实测性能提升显著。
MobileMoE 是一系列专为移动设备设计的混合专家（MoE）语言模型，活跃参数在0.3B到0.9B之间，总参数1.3B到5.3B。研究团队首次提出了面向设备端的MoE缩放定律，在内存和计算约束下优化架构，找到了中等稀疏度与细粒度共享专家的最佳平衡点。模型采用四阶段训练流程（预训练、中训练、指令微调、量化感知训练），全部基于开源数据集。在14个基准测试中，MobileMoE以2-4倍更少的推理FLOPs匹配或超越领先的密集模型，并以最多60%更少的参数超越SOTA MoE模型OLMoE-1B-7B。在商用智能手机上，MobileMoE-S在INT4权重内存下比密集基线MobileLLM-Pro实现1.8-3.8倍更快的预填充和2.2-3.4倍更快的解码。
MobileMoE: Scaling On-Device Mixture of Experts
Mixture-of-Experts (MoE) has become the de facto architecture for hundred-billion-parameter language models, yet its advantages at sub-billion scales for on-device deployment remain largely unexplored. To close this gap, we present MobileMoE, a family of on-device MoE language models with sub-billion active parameters (0.3-0.9B active and 1.3-5.3B total) that establish a new Pareto frontier for on-device LLMs. We first formulate an on-device MoE scaling law that jointly optimizes MoE architecture under mobile memory and compute constraints, identifying an on-device sweet spot - moderate sparsity with fine-grained and shared experts - that is simultaneously memory and compute-optimal. Building on the derived architectures, we train MobileMoE with a four-stage recipe covering pre-training, mid-training, instruction fine-tuning, and quantization-aware training, all on open-source datasets. Across 14 benchmarks, MobileMoE matches or exceeds leading on-device dense LLMs with 2-4$\times$ fewer inference FLOPs, and matches or surpasses the state-of-the-art MoE OLMoE-1B-7B with up to 60% fewer parameters. To bridge the last mile to mobile deployment, we provide the first efficient MoE inference on commodity smartphones with comprehensive on-device profiling. At comparable INT4 weight memory, MobileMoE-S delivers $1.8$-$3.8\times$ faster prefill and $2.2$-$3.4\times$ faster decode than the dense baseline MobileLLM-Pro.