Length-Extrapolatable Recurrent Models
Learning Length-Extrapolatable Recurrent Models
CST enables recurrent models to handle contexts 128x longer than their training length, solving a fundamental limitation in long-context AI systems.
The paper introduces Credit Stabilization through Time (CST) to address recurrent model failures beyond training horizons.
CST rescales state-credit signals during backward propagation without altering forward computation.The method improves performance at up to 128x the training length on both controlled synthetic tasks and real data with different credit dynamics.
Learning Length-Extrapolatable Recurrent Models
Recurrent models provide a natural path to long-context modeling, yet models trained with backpropagation through time (BPTT) often fail beyond their training horizon. Classical analyses emphasize gradients that vanish or explode along temporal paths. However, dense per-token losses can still train a shared recurrent rule despite severe decay, showing that decay alone does not determine whether learning fails. We instead study state credit: the signal through which future losses reach earlier recurrent states before contributing to parameter updates. Accordingly, we intervene directly on state credit and propose Credit Stabilization through Time (CST). During backward propagation, CST locally rescales the state-credit signal to stabilize its norm without rotating the component being corrected, while leaving the forward computation unchanged. Because controlled synthetic tasks and real data exhibit different credit dynamics, we specialize CST to each regime. In both settings, CST improves performance beyond the training horizon, with gains observed at up to 128x the training length.
This report is categorized under AITOP's "Papers" desk. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include CST, BPTT.
Source attribution: curated from arXiv cs.LG (@Hanwen Jiang).