KVMem improves inference efficiency
Nice paper to improve inference efficiency. It's been a while we haven't seen good work on efficien...
KVMem enables local deployment of large models with extended context on consumer hardware, making long-running agents accessible without expensive cloud infrastructure.
KVMem addresses the problem of agent workspaces outgrowing context windows before task completion. The solution keeps overflow as paged KV state across GPU memory, host memory and NVMe. On the DeepSWE test with Qwen3.8-27B, task success improved from 43.8% under compaction to 48.4%. The system can virtualize an agent workspace up to 1M tokens on a 24GB RTX 5090 laptop, four times the model's native 256K window.
Nice paper to improve inference efficiency. It's been a while we haven't seen good work on efficien...
Nice paper to improve inference efficiency. It's been a while we haven't seen good work on efficiency. Here is why it matters: A long-running agent's workspace outgrows its context window long before the task finishes. The first approach commonly used, compaction, loses the fine-grained execution evidence. And text retrieval re-prefills content the model already processed. KVMem keeps the overflow as paged KV state instead, spread across GPU memory, host memory and NVMe. Lightweight attention-space indexes, native to the model, pick the relevant historical blocks and materialize a query-dependent view that fits inside the native context window. On the DeepSWE long-context test with Qwen3.8-27B, task success goes from 43.8% under compaction to 48.4%. The local deployment result stands out. It runs Qwen3.6/3.8-27B NVFP4 with MTP on a laptop with a 24GB RTX 5090, virtualizing an agent workspace up to 1M tokens, four times the model's native 256K window, at around 50 tokens per second. Paper: academy.dair.ai/papers/kvmem-v… 💬 4 🔄 0 ❤️ 5 👀 1490 📊 5 ⚡
