想看MLA到底怎么工作的？这篇论文用114M模型做实验，发现cKV只记内容不记位置，还能省81%缓存，归纳头也全挤在一层，挺有意思的。
论文首次对DeepSeek-V2引入的多头潜在注意力（MLA）进行全面的可解释性分析。研究者训练了一个114M参数的Transformer，在TinyStories上微调后通过SVD、线性探针等方法发现：共享低秩瓶颈cKV保留了98%的实体身份信息，几乎完全丢弃位置信息，实现了81%的KV缓存压缩。归纳头集中在第12层，而标准多头注意力中它们分布在不同层。瓶颈平均仅使用46%的容量，存在过度配置。这些结果揭示了MLA并非被动压缩，而是主动重塑了模型的内容组织、位置编码和电路结构。
Through the Bottleneck: How Multi-head Latent Attention Separates Content from Position in Language Models
Multi-head Latent Attention (MLA), introduced in DeepSeek-V2, compresses key-value pairs through a shared low-rank bottleneck (cKV), achieving 81% KV-cache reduction during inference. Despite its adoption in massive production models, no prior work has studied what information this bottleneck preserves or discards, nor how it reshapes internal transformer circuits. We present the first comprehensive mechanistic interpretability study of MLA, training a 114M-parameter transformer (pretrained on a web/code/math mixture, fine-tuned on TinyStories) and analyzing its representations through SVD, attention head taxonomy, linear probing, and a disruption-attribution analysis. Our key findings are: (1) the cKV bottleneck learns a pure content representation, preserving entity identity (98% retention) while discarding positional information, validating MLA's separation of content from position via RoPE; (2) induction heads co-locate at a single layer (Layer 12), unlike their distributed formation in standard MHA; (3) a single "semantic hub" layer (Layer 15) simultaneously exhibits the highest SVD effective rank and strongest disruption-attribution score; and (4) the bottleneck is globally over-provisioned, using only 46% of its capacity on average. These findings suggest MLA does not merely compress attention passively, but reshapes how the model organizes content, position, and circuit structure. We view this as an initial data point and detail scope limitations in Section 5.