SAM与Muon结合的谱范数鲁棒性优化
Sharpness-Aware Minimization and Muon: Robustness under the Spectral Norm
这个研究把SAM的内扰动改成矩阵谱范数，再用Muon做外更新，在ImageNet的ViT和ResNet上准确率都最高。
Sharpness-Aware Minimization（SAM）通过鼓励对参数扰动不敏感来提升泛化，但扰动几何依赖尚未清晰。
本文借鉴Muon优化器的矩阵感知思路，为矩阵权重引入逐层谱内扰动，并与AdamW/SGDW或Muon外更新组合。在ImageNet-1K上对ViT-Small/16和ResNet-50的实验表明，谱内步+Muon外步在两类模型上均取得最高验证准确率。
该工作揭示了谱范数几何在SAM中的有效性。
Sharpness-Aware Minimization and Muon: Robustness under the Spectral Norm
Sharpness-Aware Minimization (SAM) aims to improve generalization by encouraging insensitivity to small, worst-case parameter perturbations. However, the notion of a "small" perturbation is inherently geometry-dependent: while existing SAM variants have explored a wide range of choices, a clear perspective on which geometries are most effective in practice remains elusive. Recent work on matrix-aware optimization, particularly the Muon optimizer, suggests that respecting the matrix structure of hidden-layer weights can lead to strong empirical performance. Motivated by this, we study matrix-aware geometry in both stages of SAM: we introduce a layerwise spectral inner perturbation for matrix-valued hidden-layer parameters and combine it with either AdamW/SGDW or Muon in the outer update. Across ImageNet-1K experiments on ViT-Small/16 and ResNet-50, we find that the combination of a spectral inner step with a Muon outer step performs consistently strongly, achieving the best validation accuracy on both models among the evaluated methods.
This report is categorized under AITOP's "Models" desk. Key entities and technical topics under continuous tracking include SAM, Muon, ImageNet-1K, 谱范数.
Source attribution: curated from arXiv cs.LG (@Wenzhi Zhong, Edward Milsom, Michael Murray).
