Muon 优化器跟 LoRA 本来不搭，这篇论文用数学近似硬凑出了个 sMuon，微调性能有提升，实现还只用矩阵乘法。
Muon 优化器在神经网络预训练中优于常见替代方法，但在 PEFT（参数高效微调）中很少使用。原因在于 LoRA 的低秩参数化无法直接对权重更新做正交化。作者提出 sMuon，通过线性化和最小二乘近似松弛的 Muon 目标，且实现只用 matmul 运算。在 SFT 和 ReLoRA 预训练实验中，sMuon 相比基线有适度性能提升。
Approximate Muon with low-rank adapters
The Muon optimizer shows clear benefits versus alternatives when pretraining neural networks. However, it is used less frequently for parameter-efficient fine-tuning (PEFT). One potential reason is that the most common PEFT method, LoRA, does not naturally combine with Muon since it is not mathematically possible to orthogonalize the weight update given by a low-rank parameterization. In this paper, we address this issue by approximating the solution to a relaxed Muon objective in the low-rank setting via linearization and then least-squares. We provide an efficient implementation that uses matmul operations only, as opposed to more complex linear algebra decomposition routines. Our method, sMuon (small Muon), performs favourably across SFT and a ReLoRA pretraining experiment. While results are model- and eval-dependent, we find overall that using Muon for low-rank fine-tuning provides moderate performance improvements.