这篇论文用理论和实验告诉你，离线RL里悲观多少不重要，悲观的结构对不对才关键，还给出了数据增强的正确用法。
论文证明在上下文MDP中，过度悲观不会阻碍最优泛化，但必须尊重最优解的对称性。通过理论分析，适度悲观但非对称的值函数可能比过度悲观且对称的值函数泛化更差。数据增强应通过在策略提取时施加一致性损失来提升泛化，而非在增强数据集上常规训练。使用IQL和CQL在旋转对称reacher环境中验证了该方法。
Generalization in offline RL: The structure is more important than the amount of pessimism
While pessimism counteracts overestimation bias in offline reinforcement learning (RL), being overly conservative has been associated with hindering certain forms of generalization. However, in this paper we demonstrate that being overly pessimistic does not inherently prevent optimal generalization in contextual MDPs (CMDPs). Instead, we argue successful generalization depends not on the amount of pessimism, but whether the pessimistic structure respects the underlying symmetries of the optimal solution. We prove that a mildly pessimistic, non-symmetric value function can generalize worse than an overly pessimistic, symmetric one. In offline RL, the structure of the pessimism is determined by the structure of the dataset coverage. As such, enforcing a symmetric value function can be non-trivial, and might require techniques such as data augmentation (DA). Inspired by our theoretical results, we argue that DA can best be applied through a consistency loss during policy extraction, rather than the common practice of (regular) offline training on an augmented dataset. This is empirically validated using IQL and CQL on a rotationally symmetric reacher environment.