Meta and MIT paper shows looping transformer layers boosts efficiency
Looped layers offer a way to trade compute for depth without inflating parameter counts, which matters for teams deploying models under strict memory or latency budgets. Practitioners building small or edge-deployed models can use the paper's findings on loop placement and lightweight adapters to squeeze more quality from a fixed architecture. Compared with simply doubling depth at 51.4% more parameters, adapters at 1.1 to 4.3% are a far cheaper path to gains.
A new Meta and MIT paper finds that reusing a transformer's layers in loops can outperform a plain model of the same size. The researchers trained dense models up to 1B parameters on up to 155B tokens, testing loop placement and repetition counts. They added per-loop adapters costing only 1.1 to 4.3% extra parameters, compared with 51.4% for doubling depth. Looping the entire stack gave the best quality at a fixed size, while looping middle layers worked best when training compute was limited. A repaired parallel looping method also lets loops run simultaneously during decoding.
New Meta and MIT paper finds reusing a transformer's layers in loops can beat a plain model.
Looping runs the same layers several times, giving a model more depth without more parameters. But each loop costs compute and normally runs in sequence for every token.
They trained dense models up to 1B parameters on up to 155B tokens, testing where to loop and how many times. They also added tiny per-loop adapters and repaired a parallel looping method so loops can run at once during decoding.
Looping the whole stack gave the best quality at a fixed size, while looping middle layers paid off best when training compute was tight. Per-loop adapters added only 1.1 to 4.3% more parameters, versus 51.4% for doubling depth, and helped more as loops increased.
