这篇论文展示了对比强化学习通过动作块建模获得显著性能提升，揭示了动作块携带更多目标信息的新机制。
研究者将对比强化学习(CRL)从单步动作扩展到动作块，在18个离线环境和11个在线环境中分别实现了31.7%和93.1%的性能提升。研究发现，在CRL中，动作块比单步动作携带更多关于目标的信息，显著提升了评价器的表示能力。这种改进并非完全通过建模非马尔可夫、时间扩展策略或传播无偏多步回报来实现。
Three Steps at a Time: Learning Representations from Action Sequences in Contrastive RL
While self-supervised approaches to reinforcement learning have achieved strong results by learning representations of states and actions, a key open question is the time scale over which actions should be modeled. Departing from the standard formulation relying on single-step actions, we extend contrastive reinforcement learning (CRL), a prototypical self-supervised method, to operate over action chunks, and find that this results in large, pervasive gains across established offline and online benchmarks: +31.7% and +93.1% across 18 and 11 environments respectively. While action-chunking-driven gains are generally explained through the ability to model non-Markovian, temporally extended policies, and to propagate unbiased multi-step returns, interestingly, we find that these arguments only partially apply to CRL. Our empirical studies suggest that, in the context of CRL, an action chunk carries more information about the goal than a single action, measurably improving the critic's representations, and rendering the algorithm significantly more effective.