Amazon SageMaker AI 实时端点新增 KV cache 感知路由
Amazon SageMaker AI real-time endpoints now support KV cache aware routing
在 SageMaker 上自托管 LLM 的朋友可以看看:新路由会把请求送到缓存命中最多的实例,长 prompt 场景能省下不少重复计算。文章还对比了它和 prefix-aware routing 的适用场景,并给了启用和验证步骤。
Amazon SageMaker AI 宣布其实时推理端点支持 KV cache 感知路由。该策略会把每个请求路由到已缓存与其 prompt 最长匹配内容的实例上,从而复用已有的 KV cache,降低推理延迟与计算开销。
工作原理是在请求进入时计算 prompt 与各实例缓存前缀的匹配长度,选择匹配最长者。这与此前提供的 prefix-aware routing 不同:prefix-aware 路由按相同前缀分组流量,而 KV cache 感知路由直接面向缓存命中量做决策,适合多租户、共享前缀较少或缓存模式多变的负载。
开发者可在单模型端点和 inference component 端点上启用该功能,并按官方文档验证路由与缓存命中效果。对于在 SageMaker 上自托管 LLM 的团队,这是不换推理框架就能提升吞吐与首 token 延迟的选项。
Amazon SageMaker AI real-time endpoints now support KV cache aware routing
Amazon SageMaker AI real-time endpoints now support KV cache-aware routing, a new strategy that routes each request to the instance holding the longest cached match for its prompt. Learn how it works, when to use it compared with prefix-aware routing, and how to enable, invoke, and verify it on single model and inference component endpoints.
This report is categorized under AITOP's "Products" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include Amazon SageMaker AI, KV cache, LLM inference, AWS.
Primary source verification: published directly by AWS Machine Learning Blog with first-party authority.