这篇论文把物理建模和DRL结合起来控无人机，SAC和TD3比PPO表现好，有具体数字和对比，搞控制或强化学习的可以看看。
论文提出一种物理感知的端到端深度强化学习(DRL)方法，直接对四旋翼的低层体轴输入（总推力与体轴扭矩）进行控制，结合高保真Simulink环境与12状态刚体模型。方法包含基于Moore-Penrose伪逆的Action2RPM分配，以及一阶作动器动力学（时间常数0.076s）和转子陀螺耦合。在四种DRL算法（DDPG、TD3、PPO、SAC）的两阶段任务中，SAC和TD3展现出更优的稳定性和探索效率，PPO样本效率较低。研究强调了建模作动器滞后与气动力矩对稳定低层控制的重要性，并提供了可复现的基准。
Physics-Aware End-to-End Deep Reinforcement Learning for Quadcopter Control with Actuator Dynamics
Unmanned aerial vehicles (UAVs), particularly quadcopters, present unique challenges for autonomous control due to their underactuated dynamics: only four available control inputs must govern six degrees of freedom. This paper investigates a physics-aware, end-to-end deep reinforcement learning (DRL) approach that acts directly on low-level body inputs, total thrust and body torques $(T, τ_x, τ_y, τ_z)$, and closes the loop through a high-fidelity Simulink environment. Our simulator integrates a 12-state rigid-body model (MATLAB Level-2 S-Function) with (i) an Action2RPM allocation based on the Moore-Penrose pseudo-inverse of a coefficient matrix derived from thrust and drag terms, and (ii) first-order actuator dynamics for each motor (time constant $T_m = 0.076$ s), including rotor gyroscopic coupling. A shaped reward balances goal-reaching and stability using an exponential position well, attitude penalties, and quadratic velocity costs. Four DRL algorithms, DDPG, TD3, PPO, and SAC, are evaluated in two stages: (S1) thrust-only hover and (S2) hover with pitch torque and a translated goal. Results show that SAC and TD3 achieve superior stability and exploration efficiency, while PPO is less sample-efficient. The study highlights the significance of modeling actuator lags and aerodynamic moments for stable low-level control and provides a reproducible benchmark for quadcopter DRL.