ParaTTT Parallelizes Deep Test-Time Training Memory Layers
ParaTTT: Parallelizing Test-Time Training Layers with Deep Memories
The core insight is that deep nonlinear test-time memories no longer have to trade expressiveness for parallelism, since PTTT recovers exact online gradient descent as a fixed point while running chunkwise in parallel. Researchers building subquadratic sequence models should examine the algorithm and benchmarks, since it directly compares against Gated DeltaNet and Mamba-2 at matched model sizes. The paper appears on arXiv as 2610.38255v1.
ParaTTT (PTTT) is a new algorithm that parallelizes test-time training (TTT) layers, which use small neural networks trained online by gradient descent as memory in recurrent architectures. Existing implementations like TTT-MLP rely on sequential block-wise weight updates, running several times slower than chunkwise-parallel linear recurrences such as Gated DeltaNet and Mamba-2. PTTT iteratively refines estimates of the inner-loop gradient trajectory using three chunkwise-parallel passes per refinement, with the exact online recurrence as a fixed point; the authors prove geometric convergence for a two-layer variant. In practice, two refinement steps at 4K context evaluate the inner-loop trajectory 32x faster than TTT-MLP at matched error, and the full 340M model trains 4.9x faster. Models at 340M and 1.3B parameters match or exceed Gated DeltaNet and Mamba-2 on language-modeling perplexity and zero-shot accuracy, with better recall.
ParaTTT: Parallelizing Test-Time Training Layers with Deep Memories
Test-time training (TTT) layers replace the linear fast-weight memory of modern recurrent architectures with a small neural network trained online by gradient descent. This nonlinear memory is more expressive than the linear memories of Gated DeltaNet and Mamba-2, but is difficult to parallelize: existing implementations such as TTT-MLP rely on sequential, block-wise weight updates that run several times slower than chunkwise-parallel linear recurrences. Moreover, their block-wise execution only approximates the token-by-token online recurrence, whereas linear recurrences admit an exact chunkwise-parallel evaluation. We introduce ParaTTT (PTTT), an algorithm for parallelizing arbitrarily deep nonlinear memories by iteratively refining estimates of the inner-loop gradient trajectory. Starting from a guess of the hidden-layer gradients, each refinement consists of three chunkwise-parallel passes: a forward pass that evaluates the resulting feature trajectory, an exact gated-delta-rule pass that computes the readout trajectory and its feature gradients, and a backward pass that propagates these gradients into a new estimate. The exact online gradient descent recurrence is a fixed point of the iteration, while decoding requires no refinement iterations. We prove geometric convergence for a two-layer variant under sufficient contraction conditions. In practice, two iterations suffice. With two refinement steps at 4K context, PTTT evaluates the inner-loop trajectory $32\times$ faster than TTT-MLP with the block approximation at matched error, while the full 340M model trains $4.9\times$ faster. Models trained at 340M and 1.3B parameters match or exceed Gated DeltaNet and Mamba-2 and remain competitive with same-size Transformers on language-modeling perplexity and zero-shot accuracy, with better recall than the linear recurrences.