看完这篇论文，你就明白怎么让 LLM 在有多层记忆时不再反复重算——InferScale 直接把 KV 缓存存好，做到检索量再大 TTFT 基本不变。
InferScale 提出一种 GPU 原生的 KV 缓存注入方案，将每个记忆事实的 KV 表示预计算并存储在 GPU 上，服务时直接注入 vLLM 的页式缓存，避免重复预填。他们引入 Chunked RoPE 解决动态组装记忆的位置编码问题，并用 Context-Window Encoding 弥补独立编码缺失的跨事实上下文。在 LoCoMo 基准上，k=50 时 TTFT 降低 72-79%（3.6-4.8x），吞吐量提升 3.7-4.5x，准确率（60.3%）接近带服务时重计算的 Mem0（63.3%）。无需修改引擎或微调模型即可集成到 vLLM。
InferScale: GPU-Native KV Injection for Personalized LLM Serving
Large language models are increasingly deployed with persistent personalized context, such as accumulated memory profiles or long conversation histories, that is shared across a user's many requests. Production memory systems (e.g., Mem0, MemGPT, and Zep) retrieve a relevant subset of this memory and inject it into the prompt, forcing the serving engine to repeatedly prefill the same content. As the retrieval budget grows, time-to-first-token (TTFT) increases even though the underlying memory is reused across requests. We present InferScale, a GPU-native LLM memory system that replaces repeated prompt prefilling with reusable KV state. InferScale precomputes each memory fact's KV representation, stores it alongside a semantic embedding on the GPU, retrieves relevant facts at serving time, and injects their KV directly into vLLM's paged cache. To support dynamically assembled memories under rotary position embeddings, we introduce Chunked RoPE, which stores keys before rotation and applies their serving-time positions during injection. However, encoding memory facts independently omits the cross-fact context available during joint prefilling. We mitigate this with Context-Window Encoding, which encodes each memory fact together with a small window of preceding conversation context while caching only the target fact's KV. InferScale is implemented through vLLM's KV-connector interface, requiring neither engine modifications nor model fine-tuning. Across three open-weight models on LoCoMo, InferScale keeps TTFT nearly constant as the retrieval budget increases: at k=50 it reduces TTFT by 72-79% (3.6-4.8x), achieves 60.3% accuracy versus 63.3% for Mem0 without serving-time recomputation, and delivers 3.7-4.5x the throughput under concurrent load. Reusable KV state thus decouples memory-conditioned serving latency from retrieved-context size while preserving application quality.