MiniCPM-V 4.7 runs 35B MoE on phones at 20.51 tokens/s
Smarter scheduling, not a smaller model, is what makes a 35B-parameter MoE fit and run fast on consumer phones — no retraining required, since routing stays untouched. Developers building on-device multimodal apps can grab the code and weights from GitHub and Hugging Face to run large models where cloud calls are impractical. It also shows open-source Chinese labs pulling ahead on mobile inference efficiency, a space Apple and Qualcomm stack owners care about.
OpenBMB's MiniCPM-V 4.7, a 35B-parameter MoE model with 3B parameters activated per token, now runs on-device at 20.51 tokens/s on an iPhone 18 Pro Max. The speed comes from Lookahead Expert Scheduling (LES), which prefetches experts via cross-layer routing prediction and overlaps loading with compute. Weight repacking into contiguous storage plus layer-wise mixed-precision quantization cuts transfer costs, and an adapted DSpark speculative sampler joins the pipeline. Compared with a 3.38 tokens/s on-demand loading baseline, the full system is 6.07x faster while keeping the original architecture and expert routing intact.
📱 A 35B MoE model running on a phone at over 20 tokens/s.
Meet Lookahead Expert Scheduling (LES), the inference technology behind MiniCPM-V 4.7’s efficient on-device performance.
🧠 The challenge: 35B total parameters, 3B activated per token, and limited device memory. Keeping experts in flash storage makes the model fit, but loading their weights can leave computation waiting.
LES co-optimizes expert loading, compute scheduling, and weight access:
⚡ Expert prefetch and compute scheduling: Cross-layer routing prediction prefetches experts the next layer is likely to need. Experts selected by the actual router that are already loaded compute first, while the remaining required experts load in parallel. Prediction only guides prefetch and scheduling: every required expert is still loaded and used. No additional training is needed.
💾 Weight access optimization: Repack expert weights into contiguous storage to reduce scattered reads, and apply precise layer-wise mixed-precision quantization to reduce weight storage and transfer costs while preserving model capability.
We also adapted DSpark speculative sampling to work alongside LES in the on-device inference pipeline.
In our iPhone 18 Pro Max tests, the full system reaches 20.51 tokens/s, up from a 3.38 tokens/s on-demand loading baseline: 6.07× faster. 🔗 https://t.co/kFD3X4RN0P
The original model architecture and expert routing stay intact. Smarter scheduling makes larger models practical on-device.
👉 MiniCPM-V GitHub: https://t.co/msjm2QlrvX 🤗 Model info: https://t.co/W95QKGhymw 💻 Technical Details: https://t.co/6Ul7qXCgjs
#MiniCPM #OnDeviceAI #OpenSource
