Papers

FrontierReplay:混合架构模型的无切分精确前缀缓存方案

FrontierReplay: Split-Free Exact Prefix Caching for Hybrid Language Models

Why we picked it

做推理服务或部署混合架构模型(比如线性注意力+循环层这类)的团队该看看。它不用切分 prefill 就能在可预测的边界处建精确缓存,TTFT 最高降 8.9 倍,精度一点不掉,比近似 replay 靠谱。

一篇系统论文提出 FrontierReplay,针对混合架构模型(自注意力加循环层)的前缀缓存难题。现有细粒度精确缓存必须在 prefill 前向传播中逐检查点切分,调度开销在高负载下抵消复用收益;粗网格保存则浪费可复用前缀。

Body

核心观察是:多轮对话、agent 循环、文档问答和树搜索中的前缀复用几乎总是终止于可预先预测的结构边界,如轮次结束和已知分支点。FrontierReplay 在这些边界上直接生成精确检查点,前向传播全程不切分;它仅记录一小段循环层输入窗口,用跨层 state-only replay kernel 重建目标状态,每 token 成本比完整重算低 6-37 倍,且无需主机同步。frontier-aware 保留策略优先保留活跃锚点和已验证分支,把闲置检查点槽位借给注意力 KV 缓存。

实验覆盖多种混合架构、模型规模和工作负载:中位首 token 时间(TTFT)降低 16-89%(最高 8.9 倍),80% SLO 容量提升最高 1.73 倍,goodput 提升最高 4.26 倍。对比近似 replay 方法,goodput 最高高 1.54 倍,且保持精确任务精度,避免近似方法在长上下文上高达 17% 的精度损失。

Original · arXiv cs.LG

FrontierReplay: Split-Free Exact Prefix Caching for Hybrid Language Models

Hybrid language models combine self-attention with recurrent layers whose in-place state updates restrict prefix reuse to exact saved positions. Saving states on a coarse grid wastes reusable prefixes, whereas existing fine-grained exact caches must cut the prefill forward pass at each mid-prompt checkpoint, adding scheduler steps that outweigh the reuse gains under load. We observe that across multi-turn chat, agent loops, document question answering, and tree search, prefix reuse almost always ends at structural boundaries predictable in advance, such as turn ends and known branch points. We present FrontierReplay, a prefix caching system that materializes exact checkpoints at these boundaries without cutting the prefill. While the forward pass runs uncut, FrontierReplay logs a short window of recurrent-layer inputs and invokes a cross-layer, state-only replay kernel to build the target state 6-37x cheaper per token than full recomputation without host synchronization. A frontier-aware retention policy prioritizes active anchors and proven branches while lending dead checkpoint slots to the attention KV cache. Across diverse hybrid architectures, model scales, and workloads, FrontierReplay delivers clear gains over existing exact caches, lowering median time to first token (TTFT) by 16-89% (up to 8.9x), raising 80% service-level objective (SLO) capacity by up to 1.73x, and improving goodput by up to 4.26x (3.67x at larger scales). Compared with approximate replay methods, FrontierReplay delivers up to 1.54x higher goodput while preserving exact task accuracy, avoiding the up to 17% accuracy drop suffered by approximation on long contexts.

Industry Coordinates & Source Verification

This report is categorized under AITOP's "Papers" desk. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include FrontierReplay, prefix caching, hybrid architecture, inference optimization.

Source attribution: curated from arXiv cs.LG (@Zekun Wang, Yizhuo Wang).