这篇论文从数学上解释了TD学习的误差为啥降不下去，用扩散近似把随机性的影响说清楚了。方法派、做强化学习理论的可以看看。
本文针对带线性函数近似的时序差分(TD)学习，提出了一种随机微分方程(SDE)近似，以替代传统常微分方程(ODE)描述。ODE仅捕捉渐近平均动力学而忽略随机波动，新SDE模型在马尔可夫噪声下区分了投影Bellman算子的收缩动力学与采样噪声的影响。该模型解释了恒定步长误差下限是由马尔可夫长期协方差与投影Bellman算子收缩几何相互作用导致。论文提供了理论证明与数值验证。
Temporal difference (TD) learning with linear function approximation is a core method for policy evaluation. Its classical continuous-time description is an ordinary differential equation (ODE), which captures the asymptotic mean dynamics but neglects stochastic fluctuations determining the error floor. We introduce a stochastic differential equation (SDE) approximation for linear TD(0) under Markovian noise. The resulting model distinguishes the contraction dynamics governed by the projected Bellman operator from the influence of Markovian sampling. As a consequence, the model explains the constant-stepsize error floor through the interaction between Markovian long-run covariance and the contraction geometry of the projected Bellman operator.