做大规模 LLM 推理部署的团队，这篇论文给出了跨 GPU 注意力优化的新思路——路由查询而非移动缓存，实测能大幅降低延迟。建议关注其成本模型和决策谓词，可直接用于优化自家推理系统。
本文研究跨 GPU 实例的注意力机制优化问题。传统方法在查询需要访问其他 GPU 上的 KV 缓存块时，会移动缓存块到查询所在 GPU，但多查询注意力（MLA）将每个 token 的键和值压缩为窄向量，使得路由查询（约 1KB）比移动缓存块更便宜。作者在真实多节点 H100 集群上测量了跨实例 MLA 注意力，提出了拓扑感知成本模型和路由/获取/本地决策谓词，发现解码时路由查询可将缓存移动的约 3 毫秒开销降低到几十微秒。该模型不限于 MLA，可推广到 DeepSeek-V3.2、V4 和 GLM-5.1 等架构。
Move the Query, Not the Cache: Characterizing Cross-Instance Latent Attention Redistribution Across GPU Fabrics
Frontier LLMs increasingly decide what a query attends to with a sparse-attention indexer that picks a few KV-cache blocks per query: attention's unit is now a small, reusable chunk. Agentic workloads hammer it: many sub-agents query one large codebase, reusing the same blocks. When that corpus outgrows one GPU it is partitioned across instances, so a query and the blocks it selects often sit on different GPUs: answering it means attention across instances. The reflex of prior cross-instance KV systems is to move the cache: pull the selected blocks to the requester. Multi-head Latent Attention inverts the arithmetic, compressing each token's key and value into one narrow vector, so a routed query row is only ~1 KB, smaller than the chunk it attends; routing the query is then often cheaper than moving the cache. Which primitive wins, over which fabric and request shape, is uncharted, least of all on device-initiated RDMA that makes per-request cross-node transfers cheap. We characterize cross-instance MLA attention on a real multi-node H100 cluster, distilling two reusable artifacts: a topology-aware cost model (probe / transfer / compute / return / merge) and a closed-form route/fetch/local predicate, whose constants we measure on real IBGDA, where the model tracks batched round-trips to within ~7%. At decode it routes the query, trading the cost of moving the cache (a ~3 ms re-adaptation splice for a contiguous chunk, or a scattered gather under selection) for a tens-of-microsecond round trip, and picks the fabric by probe latency, not peak bandwidth. We instantiate the cost model and predicate for MLA, but neither is MLA-specific: they apply wherever compression or sparse selection shrinks attention to small chunks (DeepSeek-V3.2, V4, and GLM-5.1 today). Extending them to a new architecture requires measuring just two coefficients: the routed payload and fetch's move-the-cache cost.