FlatCake: Flat KVCache Storage for Faster LLM Serving
The key insight is that KV blocks are write-once and immutable, so caching hierarchies designed for general workloads actively hurt LLM serving — placing blocks on their final medium at creation outperforms migration-based designs. Engineers building or tuning inference infrastructure should evaluate flat layouts and GPUDirect Storage paths against Mooncake-style tiers, especially for agentic sessions with multi-gigabyte prefixes. The benchmark comparisons against Mooncake and Mooncake-GDS provide concrete baselines for such evaluations.