Mini 27B looping MoE model runs on 32GB machines
Looping MoE is the interesting mechanism here: it reuses a small expert slice across cycles, letting a 27B model run within 32GB without the usual quantization compromises. Developers running local inference on constrained hardware can benchmark it against dense models of similar active parameter counts. The reported 75 tok/s decode on a DGX Spark gives a concrete baseline for early performance tuning.
A new compact "mini" model has been released, using a 27B-parameter looping MoE architecture with only 3B parameters active per token. The design allows it to fit on machines with 32GB of memory, making it accessible without large server hardware.
Soumith Chintala reported running it locally on both a DGX Spark and a Mac Mini. On the DGX Spark the model currently decodes at roughly 75 tokens per second, and his agents are actively working to push that throughput higher.
A looping MoE keeps total parameters large while activating a small slice per step, which trades compute for a bigger knowledge footprint at consumer-friendly memory budgets. For local-model users and the r/LocalLLaMA community, a 27B-capable model that fits in 32GB is a notable addition to the local inference landscape.
mini model is out. 27B looping MoE, 3B active. fits on a 32GB memory machine. Been running this on my DGX Spark, Mac Mini.
On my DGX Spark, got it to run roughly at 75 tok/s decode so far (my agents are actively trying to make it go faster).
Hopefully r/localLlama will be more happy now ;-)
This report is categorized under AITOP's "Models" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows and US-China foundation model benchmarks and commercialization comparison. Key entities and technical topics under continuous tracking include MoE, open-source models, local deployment, DGX Spark.
Source attribution: curated from Soumith Chintala (PyTorch) (@soumithchintala).