FlatCake 论文提出扁平 KVCache 存储架构，缓存命中率达 94%
FlatCake: A Flat KVCache Storage Architecture for LLM Serving
这篇论文把 DRAM 和 SSD 拉平成一个地址空间，跑 Claude Code trace 时命中率 94%，比 Mooncake 高不少，做推理服务的值得看看实现思路。
FlatCake 是一篇 LLM 推理服务系统的论文，把主机 DRAM 和 NVMe SSD 视为对等介质，放进统一 64 位地址空间，取代传统的分层缓存结构。KV 块写入时直接落到目标介质并保持会话内连续、按层排列的布局，不再迁移，同时通过 GPUDirect Storage 把合并后的 SSD 读直接流入 GPU。在 Claude Code 生产 trace 上跑 DeepSeek-V4-Flash，负载升高时 FlatCake 维持 94% 缓存命中率，对比 Mooncake 的 64%，并消除了 Mooncake 6.5–6.7 倍的 SSD 读放大，峰值负载平均 TTFT 降低 1.6 倍。在 Llama-3.1-70B 的 8K–128K 上下文测试中，平均 TTFT 相比 Mooncake 最多降低 2.4 倍。
FlatCake: A Flat KVCache Storage Architecture for LLM Serving
KV cache reuse accelerates LLM serving, but existing systems manage host DRAM and NVMe SSD as a tiered hierarchy where DRAM acts as an intermediate cache. Production LLM workloads break both premises of this hierarchy. First, bimodal reuse defeats DRAM temporal filtering, since workloads are dominated either by one-shot blocks with long reuse distances or by multi-gigabyte agentic sessions that exceed DRAM capacity, putting SSD on the critical read path. Second, DRAM staging and eviction-ordered layout bottleneck SSD restoration. Holding host staging buffers for in-flight prefixes causes contention under load that aborts prefetches, while incremental eviction scatters a session's blocks across the SSD and prevents large sequential reads. We present FlatCake, a flat KV storage architecture that exposes host DRAM and NVMe SSD as peer media in a unified 64-bit address space. Exploiting the fact that KV blocks are write-once, immutable, and reused at session scope, FlatCake places each block directly onto its target medium in session-contiguous, layer-major order at creation time and never migrates it. FlatCake streams coalesced SSD reads directly into a bounded GPU window via GPUDirect Storage and decouples DRAM and SSD request queues to prevent head-of-line blocking. On a production Claude Code trace with DeepSeek-V4-Flash, FlatCake sustains a 94% cache hit rate as load increases (versus 64% for Mooncake), eliminates Mooncake's $6.5\times$--$6.7\times$ SSD read amplification, and reduces peak-load mean TTFT by $1.6\times$ while improving total token throughput by $1.6\times$. Across 8K--128K contexts on Llama-3.1-70B, FlatCake further reduces mean TTFT by up to $2.4\times$ over Mooncake and $1.8\times$ over Mooncake-GDS.
