JustFit: 200K-Token LLM Serving on a 24 GiB Laptop
JustFit: 200K-Token LLM Serving on a 24 GiB Laptop with Just-in-Time State Management
This research demonstrates a significant improvement in the practicality of running large language models locally by reducing memory requirements.
JustFit, an MLX-based inference runtime, allows the Qwen3.
8-27B model to handle 196,608 input and 16,384 output tokens on a 24 GiB M4 Pro MacBook, increasing single-request context from 30,720 to 212,992 positions.
It achieves 19.11 tokens/s for a 32K-input, 64-output probe and a median peak process footprint of 16,374 MiB.
The runtime correctly answers 29 of 30 AIME 2026 problems.
JustFit: 200K-Token LLM Serving on a 24 GiB Laptop with Just-in-Time State Management
Capable open-weight models make local coding and reasoning attractive, but their context and execution state strain laptop memory. We present JustFit, an MLX-based inference runtime that combines KVExec for compressed KV execution, PhaseSwap for component residency, and StateTrans for state-preserving serving transitions. These mechanisms fuse reconstruction and coordinate just-in-time materialization and release, independently of model-weight quantization. In full-execution capacity tests on a 24 GiB M4 Pro MacBook running Qwen3.8-27B MXFP4, three independent runs complete 196,608 input and 16,384 output tokens, increasing completed single-request context from the mlx-vlm baseline's 30,720 positions to 212,992 (6.93x); a separate two-request run retains 229,376 positions in aggregate. In separate performance tests, a 32K-input, 64-output probe reaches 19.11 tokens/s, and a repeated 32K+6K workload has a median peak process footprint of 16,374 MiB. The integrated runtime answers 29 of 30 AIME 2026 problems correctly, showing how compact state and lifetime-aware execution expand local serving capacity while supporting extended generated reasoning.
This report is categorized under AITOP's "How-to" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows and US-China foundation model benchmarks and commercialization comparison. Key entities and technical topics under continuous tracking include LLM serving, Qwen3.8-27B, MLX, inference.
Source attribution: curated from arXiv cs.AI (@Yuhua Chen). Cross-source corroboration: 2 independent sources have covered this development (including IT之家), confirming industry-wide signal strength.
- IT之家09-15 06:14Source
