做 LLM 推理优化的团队可以直接参考这个设计——砍掉一半 KV 缓存但几乎不损质量，值得在自家模型上试试。
一篇新论文发现 Transformer 的 Key 和 Value 投影可以共享同一映射，从而将 KV 缓存减少 50%，而困惑度仅上升 3.1%。最佳变体 Q-K=V 保留了 Query 的独立性，使注意力仍具有方向性。结合 GQA 和 MQA 时，缓存削减可达 87.5% 和 96.9%。弱变体 Q=K-V 因对称性不适合因果语言模型，且无缓存节省。该发现挑战了传统 QKV 三投影的必要性，对推理内存优化有重要意义。
Interesting, this paper shows that Transformers ma…
Interesting, this paper shows that Transformers may not need separate key and value projections to work well.
This paper's design cut the KV cache by 50% in language modeling with only 3.1% higher perplexity, meaning inference memory fell sharply while prediction quality stayed close.
A normal attention layer makes Query to ask what each token needs, Key to label what each token offers, and Value to carry the information sent back.
Here, the surprising result is that Key and Value can often share the same learned map, because the model can use one representation both as an address and as the content being retrieved.
The best variant, Q-K=V, kept Query separate, so attention still had direction: one token can ask a different token for information instead of every relation becoming mirror-like.
When stacked with GQA and MQA, the same idea reached 87.5% and 96.9% cache cuts, because it reduces projection storage while those methods reduce stored heads.
The weak variant is Q=K-V, because tying Query and Key makes attention too symmetric for causal language, and it gives no KV-cache savings.
----
Link – arxiv. org/abs/2606.04032v2
Title: "Do Transformers Need Three Projections? Systematic Study of QKV Variants"