这篇论文把残差连接本身做成了PEFT方法，跟LoRA搭配效果更好，如果你在做模型微调可以看看这个新思路。
该论文提出流形约束超连接（mHC），将残差连接泛化作为新型PEFT方法，在冻结的OLMo-2骨干上学习残差路由模块。实验发现微调时将残差混合矩阵固定为单位阵通常能提升性能。作为独立PEFT方法时mHC未持续优于LoRA，但在匹配可训练参数预算下，mHC+LoRA组合在1B和7B规模上改善了语言建模损失并带来任务相关性基准增益。
Manifold-Constrained Hyper-Connections for Parameter-Efficient Finetuning
Most parameter-efficient finetuning (PEFT) methods adapt weights or activations, thus leaving one of the key Transformer components unchanged: residual connections. This paper investigates Manifold-Constrained Hyper-Connections (mHC), a generalisation of residual connections, as a novel PEFT approach, wrapping frozen OLMo-2 backbones with learned residual routing modules. We find that mHC can finetune frozen Transformers, but that its role differs fundamentally from the original pre-training setting: in finetuning, fixing the residual mixing matrix to identity often improves performance. As a standalone PEFT method, mHC does not consistently outperform LoRA. However, at matched trainable parameter budgets, mHC+LoRA combinations improve language-modelling loss and show task-dependent benchmark gains at both 1B and 7B scale. Overall, our results identify residual routing as a distinct and promising novel PEFT axis.