#推理基础设施

Shared KV-cache storage turns context reuse into an infrastructure layer rather than a per-node optimisation, which could cut memory costs for large inference fleets. Operators building large-scale serving stacks on Huawei NPUs should evaluate it against keeping KV caches in HBM or node-local SSDs. The 60-microsecond hop figure is the number to benchmark against competing approaches.

官方一手pandaily@[email protected] (Pandaily)Source