这篇论文给自改进对齐的收敛性补上了理论证明，SAIL-RevKL加了逆向KL惩罚后效果更好，对做LLM对齐和强化学习的朋友很有参考价值。
本文分析了自改进对齐（SAIL）算法在解决分布偏移问题时的收敛性质，指出原始SAIL目标函数因Hessian性质不保证强凹性。作者提出正则化目标SAIL-RevKL，引入逆向KL散度惩罚改善优化景观，并证明该目标在有限参数空间内满足Polyak-Lojasiewicz（PL）条件。理论贡献包括建立全局收敛保证，实现近线性样本复杂度。实验表明，SAIL-RevKL在MuJoCo基准和LLM对齐任务上均优于原始SAIL。
On the Convergence of Self-Improving Online LLM Alignment
The Self-Improving Alignment (SAIL) algorithm addresses distribution shift by reducing a bilevel formulation of the problem to an efficient, single-level method. Empirically, SAIL has demonstrated strong performance on this task. However, a formal analysis of its convergence properties has been lacking. We identify a key theoretical challenge: the standard SAIL objective function is not guaranteed to be strongly concave due to unfavorable properties of its Hessian. To address this limitation, we propose a regularized objective, SAIL-RevKL, which incorporates a reverse Kullback-Leibler (KL) divergence penalty to improve the optimization landscape. Our central theoretical contribution is to prove that this regularized objective satisfies the Polyak-Lojasiewicz (PL) condition within a bounded parameter space. We establish global convergence guarantees, achieving a near-linear sample complexity. We further validate the effectiveness and stability of SAIL-RevKL through empirical evaluations, demonstrating that it outperforms the vanilla SAIL on both MuJoCo benchmarks and LLM alignment tasks.