GSBR：面向在线强化学习的几何支持行为正则化方法
GSBR: Geometric Support Behavior Regularization for Online RL
机器人 RL 训练的新正则化方法，19 个任务上把成功率从 33% 拉到 95%，代码已开源，做机器人策略优化可以看看。
论文提出 GSBR（Geometric Support Behavior Regularization），用于在线 RL 中对预训练基础策略的正则化。与行为克隆正则化和硬支持约束不同，GSBR 按动作到基础策略支持集的距离施加惩罚：支持集内不惩罚，支持集外惩罚随距离平滑增长，且当价值收益超过惩罚时仍允许策略改进。该方法只需对基础策略的样本访问即可估计距离，基础策略通常是扩散模型或 flow-matching 模型。在 4 个机器人基准的 19 个任务上，GSBR 将平均成功率从基础策略的 33% 提升到 95%，归一化吞吐量比次优 RL 方法高约 23%。
GSBR: Geometric Support Behavior Regularization for Online RL
Regularizing online reinforcement learning (RL) toward a pretrained base policy can improve sample efficiency and learning stability, but regularization that is too restrictive limits how much the policy can improve. Base policies -- typically diffusion or flow-matching models trained by imitation learning on large, diverse datasets -- capture many behaviors that are reasonable but suboptimal. Existing regularizers using such base policies are too restrictive in one of two ways: behavioral cloning regularizers pull the actor toward the full base-policy distribution, penalizing it for discarding suboptimal modes, while hard support constraints forbid improvements that lie just outside the base policy's support. We propose Geometric Support Behavior Regularization (GSBR), which penalizes actions by their distance from the base policy's support. Actions within the support incur no penalty, so the actor can concentrate on high-value modes; outside the support, where value estimates are less reliable, the penalty grows smoothly with distance while still permitting improvement when the value gained outweighs the penalty. GSBR estimates this distance from a finite set of base-policy samples, so it requires only sample access to the base policy. Across 19 tasks on four robotic benchmarks, GSBR raises the average success rate from 33% for the base policy to 95%, achieving the highest average success rate and throughput among all compared methods, with about 23% higher normalized throughput than the next-best RL method. Project website and open-source code at https://charlesxu0124.github.io/gsbr/ .