研究显示 Adam 等自适应优化器会让策略排序出错，提出双跑修复法
Adaptive Optimizers Can Get a Policy Ranking Wrong: Momentum, Dependent Data, and a Two-Run Remedy
做强化学习调参的值得看：论文说 Adam 固定平均率会把策略价值估偏，排序都排错，还给了个双跑修法，多跑一两次拟合就能纠正。
arXiv 论文（编号 2611.01216）指出，在用线性 TD 学习估计策略长期价值时，Adam 这类自适应优化器的固定平均率会改变估计对象本身，导致拟合偏离目标 TD 值。这种偏移在不同策略间不同，无法在比较中相互抵消，数据越多反而让错误排序显得更确定。作者提出的修复方法是在另一个平均率下对同一批数据再拟合一次，用两倍新拟合减去原拟合即可消除主要偏移项。在纽约出租车和电池调度这两个用真实数据校准的模型中，该修复几乎纠正了所有错误的策略排序，代价是额外一到两次拟合。
Adaptive Optimizers Can Get a Policy Ranking Wrong: Momentum, Dependent Data, and a Two-Run Remedy
Operators choosing among decision rules, such as when a battery charges or where an idle taxi goes, rank them by estimated long-run value. A standard estimator, linear temporal-difference (TD) learning, fits values from one stream of correlated observations. Large fits often use an adaptive optimizer, such as adaptive moment estimation (Adam), with fixed averaging rates. We show that these rates can change what is estimated, not just how fast: the same observations drive the updates and the optimizer's step-size scaling, moving the fit away from the intended TD value. The shift differs across rules, so it need not cancel in a comparison, and more data only makes a wrong ranking look more certain. Our remedy refits the same data at a second averaging rate that halves the effective gain, the number to which the leading shift is proportional. Twice the new fit minus the original removes that shift. In New York taxi and battery models calibrated to real data, common optimizer settings leave the selected rule unchanged but order close rules wrongly and misstate its value. Our remedy corrects nearly every wrong ordering, with one extra fit when the shift is moderate and two when it is larger.