做稀疏模型训练或Transformer优化的研究者，HORST直接解决了AdamW在高稀疏度下效果差的问题，值得在实验中尝试替换优化器。
HORST是一种新型优化器，通过组合自适应优化器的L∞稳定性和L1稀疏偏置，解决了稀疏Transformer训练中稳定性和稀疏性难以兼得的问题。它利用非交换算子组合优化几何，基于双曲镜像映射实现鲁棒稀疏训练。实验表明，HORST在视觉和语言任务的Transformer稀疏训练中，在所有稀疏度水平上均显著优于AdamW基线，尤其在高稀疏度下提升巨大。这项工作为高效稀疏模型训练提供了新思路。
HORST: Composing Optimizer Geometries for Sparse Transformer Training
Sparsifying transformers remains a fundamental challenge, as standard optimizers fail to simultaneously encourage sparsity and maintain training stability. Effective adaptive optimizers exhibit an implicit $L_{\infty}$ bias favoring stability, yet, sparsity requires an $L_1$ bias. To integrate sparsity, we propose a composition of optimizer steps, which we cast as non-commutative operators to analyze and combine their optimization geometry in a principled way. This yields HORST (Hyperbolic Operator for Robust Sparse Training), a modular optimizer that inherits stability from adaptive methods while inducing $L_1$ sparsity bias through a hyperbolic mirror map. Our experiments demonstrate its utility for sparse training of transformers on both vision and language tasks. HORST consistently and significantly outperforms AdamW baselines across all sparsity levels, with large gains at higher sparsity.