Grid Cells as Spectral Successor Features: A Reusable Basis for Continual RL
Grid Cells as Spectral Successor Features: A Reusable Basis for Continual Reinforcement Learning
论文把网格细胞的谱性质做成了持续强化学习的特征基,新任务只跑一次线性回归,不用训练新网络也不用回放。对比 SimpleSF 和 DQN 在目标切换后回报崩掉,它的方法全程接近最大回报。做持续学习或 RL 研究的人值得照着公式复现一遍。
一篇 arXiv 论文提出用固定参考策略转移矩阵的特征向量作为后继特征(successor features)的基,灵感来自内嗅皮层网格细胞与后继表示特征向量的对应关系。
在该基下,某状态-动作对的后继特征可表示为同一处基特征的线性组合,系数有闭式解:每个系数为 θ_i = 1/(1-γλ_i),其中 λ_i 是对应特征值。这带来三点简化:长期回报与即时奖励用同一组特征预测,无需后继特征网络或经验回放缓冲区,适配新任务只需对奖励做一次线性回归。
智能体在参考策略的动作价值上做贪心选择,即对参考策略做一步策略改进,足以到达单一目标。在双目标 gridworld 实验中,以均匀随机游走(其特征向量为低频 Laplacian 特征向量)作参考策略时,该方法在每次目标切换后都保持接近最大回报,而 SimpleSF 和 DQN 每次切换都损失大部分回报。
Grid Cells as Spectral Successor Features: A Reusable Basis for Continual Reinforcement Learning
Continual reinforcement learning requires agents to adapt when the task changes while reusing what they know about the environment. Successor features separate dynamics from task-specific reward, but their effectiveness depends on the choice of base features. Taking the identification of entorhinal grid cells with eigenvectors of the successor representation as a design principle, we build this basis from eigenvectors of the transition matrix of a fixed reference policy. With such a basis, the successor features at a state-action pair are a linear combination of the base features at that same pair, which is not true in general, and the coefficients are known in closed form: each successor feature is its base feature multiplied by $θ_i = 1/(1-γλ_i)$, where $λ_i$ is its eigenvalue. The long-term return is thus predicted from the same features as the immediate reward, no successor-feature network or replay buffer is needed, and adapting to a new task reduces to a linear regression on the reward. The agent acts greedily on the action values of the reference policy, which is one step of policy improvement over it and suffices to reach a single goal. In a two-goal gridworld, with the uniform random walk as reference policy, whose eigenvectors are the low-frequency Laplacian eigenvectors, agents built this way maintain near-maximal return across every change of goal, whereas SimpleSF and DQN lose most of their return at each change.
This report is categorized under AITOP's "Papers" desk. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include reinforcement learning, Successor Features, continual learning, papers.
Source attribution: curated from arXiv cs.LG (@Élodie Côté-Gauthier, Doina Precup, Isabeau Prémont-Schwarz).