这篇论文用PEARL方法解决了RL在复杂动力学系统中样本效率低的问题，能在非稳态流中做实时控制，比现有算法更高效，值得搞控制或RL的人看看。
论文提出PEARL框架，结合强化学习与最优控制，利用自动微分计算短时策略梯度，并通过神经网络近似伴随灵敏度以估计未来回报，从而减少环境交互次数。在非稳态流中的两个参数化导航问题测试中，PEARL在样本效率和泛化性上优于现有RL算法，可扩展到高维状态动作空间，无需低维表示或多智能体策略。
Physics-enhanced reinforcement learning for real-time optimal control of dynamical systems
Reinforcement learning (RL) has recently emerged as a promising feedback control strategy for nonlinear and complex dynamical systems. However, RL algorithms are sample inefficient and require a large number of interaction with the environment to synthesize optimal control strategies. Consequently, applications of RL are typically limited to sparse sensors and actuators due to the curse of dimensionality entailed by the exploration-exploitation dilemma in high-dimensional spaces. In this work, we bridge RL and traditional optimal control for dynamical system with a novel Physics-EnhAnced Reinforcement Learning (PEARL) paradigm tailored to the control of high-dimensional and parametric dynamical systems, exploiting the differentibility of their dynamics. Specifically, PEARL employs an actor-adjoint algorithm that leverages automatic differentiation to compute policy gradients over short horizons and adjoint-based sensitivities of future returns approximated via neural networks, significantly reducing the number of environment interactions, while mitigating long-term gradient instabilities. Through two challenging parametric navigation problems in unsteady flows, we show that PEARL (i) effectively exploits differentiable environments to outperform state-of-the-art RL algorithms, (ii) is sample efficient, thanks to the physics-guided policy learning, (iii) generalizes across multiple scenarios, which is crucial when dealing with parametric systems, and (iv) enables scaling RL to high-dimensional state and action spaces, without requiring low-dimensional state representations or multi-agent strategies.