谁为 KV Cache 买单？unalloc 拆解共享推理成本归属
Who Pays for the KV Cache? Attributing Shared AI Inference Spend Across Kubernetes and LLM Provider Bills
unalloc 把 Kubernetes 和 LLM API 账单拼成一本账，还量化出按 token 和按时间计费能差 12 个点。
论文提出开源工具 unalloc，把 OpenCost、LiteLLM 以及 OpenAI、Anthropic 的成本数据合并成一本精确台账，并标出无人认领的支出占比。在构造的多 pod 部署场景中，只给 LeaderWorkerSet 的 leader pod 打 owner 标签会让该部署 66% 的 GPU 账单无主，默认回退键则把 61% 的支出归到一个 Helm chart 名下，未分配比例表面降至 4%。数据源口径同样有坑：同时开启所有来源会把网关支出全部重复计算，只读一页账单 API 只能看到约四分之一的支出。在 NVIDIA H100 上跑 vLLM 时，按 token 计量比按时间份额计量让检索密集型租户多摊 12-14 个百分点的账单，而各负载下 GPU 利用率读数都停在 97-99%。论文还把这组结果与近期的 Shapley 能耗归因研究做了对照。
Who Pays for the KV Cache? Attributing Shared AI Inference Spend Across Kubernetes and LLM Provider Bills
Organizations pay for AI through disconnected ledgers: Kubernetes allocations for self-hosted inference, gateway logs, and per-token bills from API providers. We present unalloc, an open-source tool that joins OpenCost, LiteLLM, OpenAI and Anthropic cost data into one exact ledger and reports the share of spend with no owner, and use it to study where attribution breaks at the seams between these systems. Five case studies run inference for real or simulate it: a vLLM-style serving simulator with paged KV memory and prefix caching; a PyTorch transformer serving a multi-tenant trace with a real KV cache; tensor- and pipeline-parallel inference on torch.distributed; the unmodified CLI against mock provider APIs; and four downstream use cases. At the seams, in a constructed multi-pod deployment scenario -- one month of synthetic OpenCost allocations, not observed billing data -- owner labels set only on LeaderWorkerSet leader pods leave 66% of that deployment's GPU bill unowned, and the natural fallback key assigns 61% of it to a Helm chart name while the headline unallocated share falls to 4%; enabling every source double counts all gateway spend; and reading one page of a billing API reports a quarter of spend. Inside a shared inference server the metering rule decides who pays: on an NVIDIA H100 running vLLM, a token meter assigns a retrieval-heavy tenant 12-14 percentage points more of the bill than an equal time-share meter at every load tested, while GPU utilization reads 97-99% across configured loads of 2 to 16 requests per second (3.7 to 26.9 completed requests per second; the configured rate counts session-initial arrivals only) and power draw tracks load. Neither meter is a ground truth; we position these results against recent Shapley-based energy attribution. Code, raw data, captured evidence, figures and the paper regenerate from the repository.
