How-toSelected

Ollama 与 Together AI 分享保持编码智能体低延迟的前缀缓存实践经验

Why we picked it

做编码智能体或重度用 Claude Code、Codex 的开发者该看看:Together AI 和 Ollama 拆解了 prefix cache 失效的几个真实原因,比如系统提示里一个时间戳就会让缓存全废。排查延迟和账单时能直接对号入座。

Together AI 的 zainhas 与 Ollama 的 parthsareen 联合发布博客,总结编码智能体在 Claude Code、Codex、Pi 等工具中保持响应速度的缓存经验。编码智能体每一轮都会重发完整对话历史,prefix caching(前缀缓存)是长会话保持快速的关键,但缓存生效需要 harness(工具层)和推理引擎两侧配合。

Body

文章列出了几个具体陷阱:系统提示词里放一个时间戳就会破坏缓存命中;改变 reasoning effort(推理力度)参数会清空整个缓存;告诉智能体其输出被截断,反而能让它主动去读取漏掉的内容。

对自建智能体工具链的开发者来说,这篇内容直接指向推理成本与延迟优化:同样的 token 量下,缓存命中与否的账单和速度差距可以很大,值得对照自己的 harness 逐条排查。

Original · ollama

what we learned to keep coding agents fast in Claude Code, Codex, Pi, and more: – a single timestamp in the system prompt can break the cache – changing reasoning effort can wipe the whole cache – telling the agent its output was cut off will get it read what it missed Together AI @togethercompute coding agents resend the whole conversation on every turn. prefix caching keeps long sessions fast, but only if the harness and the inference engine both do their part. @zainhas from Together and @parthsareen from @ollama wrote down what each side needs to get right. together.ai/blog/making-op… 🔗 View Quoted Tweet📊 1 ⚡

Industry Coordinates & Source Verification

This report is categorized under AITOP's "How-to" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include Claude Code, Codex, Together AI, Ollama.

Source attribution: curated from ollama (@ollama).