做智能体服务的朋友看下，TokTier把长记录重复分词干掉了，vLLM下首字延迟降16%-34%，省GPU。
TokTier提出有状态分词方法，解决智能体场景下代码转录重复提交导致的分词不可复用问题。在153,951次真实智能体调用中，提示缓存命中率94.1%时，分词仍占用首token时延的64%。该方法只对追加位置附近的小窗口重新分词，并通过稳定边界检查拼接，在17个分词器家族的1.5e10次分割检查中零偏差。增量修复100K到3M字符耗时0.5至1.1毫秒，比HuggingFace快最多437倍；在vLLM下中位首token时延下降16%至34%。
TokTier makes tokenization stateful for agentic serving. Across 153,951 real agent calls with a 94....
TokTier makes tokenization stateful for agentic serving. Across 153,951 real agent calls with a 94.1% prompt-cache hit rate, tokenization eats up to 64% of time to first token. Coding agents resubmit a long transcript after every tool result, and even a short append can shift token boundaries at the tail of the previous sequence, so nothing gets reused. TokTier is a stateful tokenization service. For a session continuation it re-tokenizes a small window around the append, runs a stable-boundary check, and splices only when that check passes. When there is no reusable prefix it runs exact pre-tokenization and BPE on a GPU. Emitted token IDs always match full reference tokenization. Differential campaigns across 17 tokenizer families covered 1.5e10 split checks with zero divergence. Incremental repair takes 0.5 to 1.1 ms from 100K to 3M characters, up to 437x faster than HuggingFace. Under vLLM, median time to first token drops 16 to 34%. Four repair cores plus one GPU sustain 1,821 requests per second where a 16-core stateless front end saturates at 40. Paper: arxiv.org/abs/2607.29678 Track more trending AI papers in our academy: academy.dair.ai 💬 3 🔄 0 ❤️ 4 👀 1199 📊 3 ⚡