这篇论文打脸了“超级权重”的主张：单独训练它们反而让模型崩成随机猜，而LoRA用极少的参数就能搞定。想了解模型微调真正该关注什么，必看。
论文检验了LLM中“超级权重”（单参数移除后性能骤降）的存在性，发现这并非所有模型通用。在OLMo-1B和OLMo-7B上，单独训练100到8192个超级权重参数，精度降至随机猜测水平；即便扩展至周围36000个参数也无改善。相比之下，随机选取同层同等数量参数训练反而能提升基准。使用仅占0.16%参数的vanilla LoRA（更新注意力权重矩阵）成功微调，且施加于超级权重坐标的LoRA约束效果类似。结论指出参数重要性不等同于单独可训练性，有效微调需要基于全层的结构化分解。
Super Weights in LLMs and the Failure of Selective Training
Recent work identified Super Weights, individual parameters whose removal degrades model performance by orders of magnitude. We show that this degradation due to pruning Super Weights does not universally apply to all LLMs. Furthermore, if these parameters are so important, Super Weight-aware training should be effective. We show the opposite. Training Super Weights in isolation (100 to 8,192 parameters) drops accuracy to random-guessing levels on both OLMo-1B and OLMo-7B, and expanding to local neighborhoods of up to 36K parameters provides no improvement. The failure is specific to Super Weight coordinates: training an equal number of randomly chosen positions in the same down_proj layers instead improves over the baseline, so the collapse comes from targeting Super Weights, not from sparsity itself. Vanilla LoRA, updating every position in attention weight matrices through low-rank structure, succeeds with only 0.16% of parameters, and applying the same low-rank update to down_proj succeeds as well. A 10-seed ablation confirms that constraining LoRA updates at positions corresponding to Super Weight coordinates yields statistically indistinguishable results. These findings establish that parameter importance does not imply parameter trainability in isolation, and that effective fine-tuning relies on structured decompositions over entire layers rather than targeting individually important weights.