AWS 发布分布式 KV 缓存 agent skill,自托管 LLM 推理成本最多降 40%
Cut self-hosted LLM cost and latency with the distributed KV caching agent skill
如果你自己用 GPU 托管 LLM,多轮对话每次都重复算上下文纯属烧钱。AWS 把这套 KV 缓存配置做成了 agent skill,从编码 agent 里一句话就能让 agent 帮你把 ElastiCache 缓存搭好,官方数据显示成本省最多 40%、首 token 快最多 80%。自己搭 vLLM 服务的话可以照着这篇实操一下。
AWS 在机器学习博客发布了一套分布式 KV 缓存 agent skill,隶属 Agent Toolkit for AWS,可让编码 agent 自动评估并完成缓存配置。官方给出的数字是:推理成本最多降低 40%,time to first token 最多缩短 80%。
自托管 LLM 的痛点在于多轮对话中每一轮都要对相同上下文重复做 prefill 计算,GPU 一直在为已经算过的 token 付费。这套方案把 KV cache 存到 Amazon ElastiCache 的分布式缓存中,让多个推理实例复用已计算的上下文,命中缓存后直接跳过重复计算,从而同时压低成本和首 token 延迟。
区别于手动改造 vLLM 等推理框架的缓存层,这个 skill 是从 Claude Code 之类的编码 agent 里直接调用的:agent 会分析你的部署情况,评估缓存收益并完成 ElastiCache 的搭建。对自己部署开源模型、按 GPU 时长计费的团队来说,这是把基础设施调优工作交给 agent 执行的一次具体实践。
Cut self-hosted LLM cost and latency with the distributed KV caching agent skill
Self-hosting LLMs on GPUs means paying to recompute the same context on every turn. Distributed KV caching with Amazon ElastiCache can cut inference cost by up to 40% and time to first token by up to 80%. This post walks through the distributed KV caching skill in the Agent Toolkit for AWS, which evaluates and sets up the cache from your coding agent.
This report is categorized under AITOP's "How-to" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows and US-China foundation model benchmarks and commercialization comparison. Key entities and technical topics under continuous tracking include ElastiCache, KV cache, AWS, prompt engineering.
Primary source verification: published directly by AWS Machine Learning Blog with first-party authority.