KVMem提升长上下文推理效率
Nice paper to improve inference efficiency. It's been a while we haven't seen good work on efficien...
KVMem让Qwen3.6/3.8-27B在笔记本上运行百万tokens上下文，比传统压缩方法效果更好。
KVMem论文提出将溢出KV状态分页存储于GPU、主机内存和NVMe中。在DeepSWE测试中，Qwen3.8-27B任务成功率从43.8%提升至48.4%。该技术可在24GB RTX 5090笔记本上虚拟化100万tokens的代理工作区，是模型原生256K窗口的四倍。
Nice paper to improve inference efficiency. It's been a while we haven't seen good work on efficien...
Nice paper to improve inference efficiency. It's been a while we haven't seen good work on efficiency. Here is why it matters: A long-running agent's workspace outgrows its context window long before the task finishes. The first approach commonly used, compaction, loses the fine-grained execution evidence. And text retrieval re-prefills content the model already processed. KVMem keeps the overflow as paged KV state instead, spread across GPU memory, host memory and NVMe. Lightweight attention-space indexes, native to the model, pick the relevant historical blocks and materialize a query-dependent view that fits inside the native context window. On the DeepSWE long-context test with Qwen3.8-27B, task success goes from 43.8% under compaction to 48.4%. The local deployment result stands out. It runs Qwen3.6/3.8-27B NVFP4 with MTP on a laptop with a 24GB RTX 5090, virtualizing an agent workspace up to 1M tokens, four times the model's native 256K window, at around 50 tokens per second. Paper: academy.dair.ai/papers/kvmem-v… 💬 4 🔄 0 ❤️ 5 👀 1490 📊 5 ⚡