Asteria 让二阶优化从理论走向实用，做大模型训练的团队可以大幅降低显存压力，同时保持收敛速度优势，值得关注其开源进展。
二阶优化方法能提升大模型训练效率，但计算和存储开销大。Asteria 是一个运行时系统，通过将优化器状态动态分配到 GPU、CPU 和 NVMe 存储，并异步执行逆根计算，解决了这一瓶颈。在单 GPU 的 DGX Spark 上，它支持 1B 参数模型的二阶训练；在多节点 GH200 系统上，它降低了 7B 模型的开销和延迟，加速收敛。研究表明，二阶优化实用化的关键在于运行时层面的状态管理和同步。
Runtime-Orchestrated Second-Order Optimization for Scalable LLM Training
Second-order methods offer an attractive path toward more sample-efficient LLM training, but their practical use is often blocked by the systems cost of maintaining and updating large matrix-based optimizer states. We introduce \textbf{Asteria}, a runtime system designed to remove this bottleneck by separating second-order optimization logic from the critical GPU training path. Rather than keeping all preconditioner state on the accelerator, Asteria dynamically distributes optimizer state across GPU memory, CPU memory, and optional NVMe storage according to architectural constraints and runtime pressure. It further uses training hooks to prepare shadow states in advance, allowing expensive inverse-root computations to proceed asynchronously on the host while GPU computation continues. For distributed training, Asteria employs a bounded-staleness protocol that limits synchronization frequency while preserving optimizer effectiveness through topology-aware coordination. We evaluate Asteria on both memory-constrained and distributed training settings. On a DGX Spark platform with a single GB10 GPU and 128GB unified memory, Asteria supports second-order training for a 1B-parameter language model. On multi-node GH200 systems, it lowers visible optimizer overhead, reduces recurring latency spikes, accelerates convergence in wall-clock time, and maintains the optimization advantages of SOAP and KL-Shampoo in a 7B-parameter language model. Our results suggest that second-order LLM training can be made practical not by simplifying the optimizer alone, but by rethinking how optimizer state, background computation, and distributed synchronization are managed at the runtime level.