MiniCPM5-2B 4-bit EXL3 quantization shrinks weights to 1.61 GB
A 1.61 GB weight file that still hits ~70 tokens/s on a Tesla T4 shows how far small open models can be squeezed onto aging or low-power GPUs, no datacenter needed. Practitioners running local inference on modest hardware can pair it with ExLlamaV3 or TabbyAPI to serve on-device chat or agent backends. Compared with running the full-precision MiniCPM5-2B, this trade-off suits edge and consumer setups over maximum accuracy.
A community-built EXL3 4-bit quantization of MiniCPM5-2B reduces the quantized model weights to just 1.
61 GB.The 4.
0 bpw EXL3 version runs local inference with ExLlamaV3 and TabbyAPI.Testing on an NVIDIA Tesla T4 measured roughly 68-70 tokens per second.
The model is available on Hugging Face, with the base model also published by OpenBMB.
⚡ Making MiniCPM5-2B even more lightweight for local inference!
A community-built EXL3 4-bit quantization of MiniCPM5-2B brings the quantized model weights down to just 1.61 GB.
✨ Highlights: ⚡ 4.0 bpw EXL3 quantization for a smaller model footprint 🚀 ~68–70 tokens/s verified on an NVIDIA Tesla T4 💾 Just 1.61 GB for the quantized model weights 🛠️ Supports local inference with ExLlamaV3 and TabbyAPI
A great community contribution showing how MiniCPM5-2B can be optimized for more lightweight and accessible local AI deployments.
🤗 Model: https://t.co/rMRcOqqxFC 🤗 Base model: https://t.co/FZOMTZh3tS
This report is categorized under AITOP's "Models" desk, focusing on China's AI industry and domestic foundation model ecosystem. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows and US-China foundation model benchmarks and commercialization comparison. Key entities and technical topics under continuous tracking include MiniCPM5-2B, OpenBMB, quantization, local deployment.
Source attribution: curated from OpenBMB (@OpenBMB). Cross-source corroboration: 11 independent sources have covered this development (including Decoder, Artificial Analysis, Aravind Srinivas), confirming industry-wide signal strength.