StBR：用潜空间散度约束改进离线强化学习策略正则化
Variational Consequence-Driven Offline Reinforcement Learning
一篇离线RL的新论文，思路是不约束动作而是约束状态后果，用潜空间散度做可计算的替代，基准成绩有提升。
arXiv 论文 2610.38238 提出 Structural Behavior Regularization（StBR），针对离线强化学习中传统动作分布约束无法保证动态一致性的问题。该方法不直接匹配动作分布，而是通过策略诱导的状态后果分布施加约束。由于真实动力学模型不可得，StBR 用闭式潜空间散度作为后果分布散度的理论上界代理。在动力学敏感区域，该方法会自适应收紧行为约束，在多个离线 RL 基准上取得更好成绩。
Variational Consequence-Driven Offline Reinforcement Learning
Offline reinforcement learning (RL) typically mitigates distribution shift by imposing divergence constraints on the action distributions of the learned and behavior policies. However, this paradigm cannot ensure the underlying dynamic consistency structure: even minor action deviations can lead to drastically different transitions, inadvertently driving the agent away from the offline dataset. To address this, we explore policy constraints through their induced state-consequence distributions rather than relying on pure action-distribution matching. Nevertheless, directly enforcing such consequence-aware constraints is computationally intractable without access to the true underlying dynamics model. To bypass this issue, we propose Structural Behavior Regularization (StBR), which replaces the intractable objective with a closed-form latent divergence. Structuring this latent space to reflect transition dynamics renders the latent divergence a theoretically bounded surrogate for the true consequence-distribution divergence. By regularizing the learned policy with this latent divergence, StBR adaptively tightens the behavior constraint in dynamics-sensitive regions, effectively preventing deviations outside the offline dataset. Empirically, we demonstrate that StBR achieves superior performance across diverse offline RL benchmarks.