这篇论文用扩散模型来对齐强化学习中的不同环境动态，不用额外复杂设计就能搞定域适应，比现有方法好。
DADiff 是一种用于强化学习的扩散驱动跨域策略适应框架，解决源域与目标域动态失配问题。该方法通过扩散模型在生成下一状态时捕获源域和目标域生成轨迹差异来估计动态失配，并开发了奖励修改和数据选择两种变体以适应目标域。理论分析表明，给定策略在两个域间的性能差异被生成轨迹偏差所界定。在多种偏移环境下的实验结果表明，DADiff 优于现有方法，有效解决了动态失配。
DADiff: Diffusion-Driven Cross-Domain Policy Adaptation for Reinforcement Learning
Transferring policies across domains poses a vital challenge in reinforcement learning, due to the dynamics mismatch between the source and target domains. In this paper, we consider the setting of online dynamics adaptation, where policies are trained in the source domain with sufficient data, while only limited interactions with the target domain are allowed. There are a few existing works that address the dynamics mismatch by employing domain classifiers, value-guided data filtering, or representation learning. Instead, we study the domain adaptation problem from a generative modeling perspective. Specifically, we introduce DADiff, a diffusion-based framework that leverages the discrepancy between source and target domain generative trajectories in the generation process of the next state to estimate the dynamics mismatch. Both reward modification and data selection variants are developed to adapt the policy to the target domain. We also provide a theoretical analysis to show that the performance difference of a given policy between the two domains is bounded by the generative trajectory deviation. More discussions on the applicability of the variants and the connection between our theoretical analysis and the prior work are further provided. We conduct extensive experiments in environments with various shifts to validate the effectiveness of our method. The results demonstrate that our method provides superior performance compared to existing approaches, effectively addressing the dynamics mismatch. We provide the code of our method at https://github.com/hanyang-chen/DADiff-release