PapersSelected

MINT:用相遇时间指示器替代截断系数,改进多步 off-policy 强化学习

MINT: Meeting-time Indicators for Truncation in Multi-Step Off-Policy RL

Why we picked it

做强化学习的朋友可以看看这篇:MINT 把多步 TD 里最难调的 trace 长度超参数换成了自动的停止指示器,MuJoCo 和 Atari 100k 上跑赢基线,还不用逐环境调参,写 RL 代码或调 R2D2 类算法的人会感兴趣。

arXiv 论文 2611.24413 提出 MINT(Meeting-time INdicators for Truncation),针对多步 off-policy 时序差分方法中逐步 trace 系数乘积导致的方差控制与长时序信号的权衡问题。传统方法里最大 trace 长度是敏感超参数,MINT 用一个二元停止指示器替代:在耦合的 behavior 与 target 轨迹首次相遇前取 1,之后取 0。

Body

在真实价值函数处,该指示器的幂等性给出由 meeting-time survival 决定的方差界。实现上只需一次采样动作即可为停止规则提供 model-free 替代,无需转移模型或 mixing-time 估计,论文还分别推导了状态耦合算子及其替代算子在固定策略评估下的收敛条件。

实验覆盖 MuJoCo、DeepMind Control、Atari 100k、RWRL 和 OGBench,MINT 超过已报告的多步基线,其中 MuJoCo 主对比使用统一 cap,不需要逐任务调 cap 长度;备份时域自适应调整,相比长固定时域备份降低了 target 波动。

Original · arXiv: Google DeepMind

MINT: Meeting-time Indicators for Truncation in Multi-Step Off-Policy RL

Many multi-step off-policy temporal-difference methods handle policy mismatch through products of per-step trace coefficients, leading to a trade-off between variance control and long-horizon signal. The maximum trace length consequently becomes a sensitive hyperparameter. We propose MINT (Meeting-time INdicators for Truncation), which replaces these products with a binary stopping indicator: one until coupled behavior and target trajectories first meet, zero thereafter. At the true value function, the indicator's idempotence yields a variance bound governed by meeting-time survival. For practical implementation, a single sampled action provides a model-free surrogate for the stopping rule, requiring no transition model or mixing-time estimate. We derive separate sufficient conditions for contraction of the state-coupled operator and its practical surrogate under fixed-policy evaluation. On MuJoCo and DeepMind Control tasks, MINT outperforms the reported multi-step baselines; the main MuJoCo comparison uses a common cap without per-task cap-length tuning. We also demonstrate the effectiveness of MINT on Atari 100k, RWRL, and OGBench experiments; MINT adapts the backup horizon to reduce target variability relative to long fixed-horizon backups across tasks and dynamics.

Industry Coordinates & Source Verification

This report is categorized under AITOP's "Papers" desk. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include off-policy RL, MuJoCo, Atari 100k, papers.

Primary source verification: published directly by arXiv: Google DeepMind with first-party authority.