JustFit 让 27B 模型在 24GB 笔记本上处理 20万token
JustFit: 200K-Token LLM Serving on a 24 GiB Laptop with Just-in-Time State Management
朋友，刚看到 JustFit 这个技术，它能让 27B 的 Qwen 模型在 24GB 笔记本上处理 20万token，比之前的方法快很多，对本地开发很有用。
JustFit 是一种 MLX 推理运行时，通过 KVExec、PhaseSwap 和 StateTrans 机制，让 Qwen3.8-27B 模型在 24GiB MacBook 上处理 196,608 个输入 token 和 16,384 个输出 token，单次请求上下文从 30,720 增加到 212,992，支持更长的推理。
JustFit: 200K-Token LLM Serving on a 24 GiB Laptop with Just-in-Time State Management
Capable open-weight models make local coding and reasoning attractive, but their context and execution state strain laptop memory. We present JustFit, an MLX-based inference runtime that combines KVExec for compressed KV execution, PhaseSwap for component residency, and StateTrans for state-preserving serving transitions. These mechanisms fuse reconstruction and coordinate just-in-time materialization and release, independently of model-weight quantization. In full-execution capacity tests on a 24 GiB M4 Pro MacBook running Qwen3.8-27B MXFP4, three independent runs complete 196,608 input and 16,384 output tokens, increasing completed single-request context from the mlx-vlm baseline's 30,720 positions to 212,992 (6.93x); a separate two-request run retains 229,376 positions in aggregate. In separate performance tests, a 32K-input, 64-output probe reaches 19.11 tokens/s, and a repeated 32K+6K workload has a median peak process footprint of 16,374 MiB. The integrated runtime answers 29 of 30 AIME 2026 problems correctly, showing how compact state and lifetime-aware execution expand local serving capacity while supporting extended generated reasoning.
- IT之家09-15 06:14原文
