Recurrent Looped Transformer architecture proposed
This is a significant new architecture proposal for large language models, potentially offering improved efficiency and performance.
A new technical report proposes the Recurrent Looped Transformer (RLT), an architecture that makes the decoder recurrent over every token. It combines the token's encoder representation with its own final hidden state from the previous token and a sliding-window cache of recent activations. With a 48-layer decoder, the computation path grows with the sequence length, while per-token cost remains fixed. The report is a design proposal with goals of reasoning gains, hardware speedups, and RL scaling, though these have not been measured yet.
Looped transformers are a popular architecture topic right now. This new technical report extends the loop across tokens. Recurrent Looped Transformer (RLT) makes the decoder recurrent over every token, prompt and response included. A causal encoder builds global KV memory. For each new token, the decoder combines the token's encoder representation with its own final hidden state from the previous token and a sliding-window cache of recent activations. With a 48-layer decoder, the computation path after t tokens runs through 48t decoder blocks, while each token still executes a fixed number of blocks. Depth grows with the sequence and per-token cost stays the same. The same state transition is used for pretraining, SFT, sampling and RL replay, and nothing resets at the prompt-response boundary. RL replay rebuilds states under the current weights instead of reusing stale rollout states. The report is a design proposal. The author states that reasoning gains, hardware speedups and RL scaling are goals that have not been measured yet. Paper: github.com/yifanzhang-pro… Chat with Paper: academy.dair.ai/papers/recurre… 💬 4 🔄 2 ❤️ 12 👀 1015 📊 6 ⚡
