#looped model

1 stories · 0 in 7 days · 0 in 30 days
Stories, products and papers tagged "looped model", newest first.
Oct 6
PapersBuildersSelected13:42
Fixed-Point Training Speeds Up Looped Language Models

The key insight is that near-fixed-point recurrent states make much of the compute spent on full backpropagation and full KV caches redundant, cutting costs across training, decoding and RL at once. Practitioners building efficient inference stacks or RL pipelines for recurrent models can apply KV sharing and distilled prefill directly. Compared with Huginn's fixed prior, the learned prior plus orthogonal injection improves perplexity at every scale tested up to 1.6B parameters.

arXiv cs.LG@Benhao Huang et al. (7)Source