Tsinghua paper shows training order improves hybrid LLM architectures
The non-obvious finding is that retrieval skill in hybrids is allocated by training order, so a simple freeze-then-unfreeze schedule can outperform longer training or more data. Teams pretraining hybrid or linear-attention models can apply BLD directly at no extra token budget. Compared with architecture-level fixes, it is a scheduling change, making it cheap to test in existing pipelines. Chinese-language sources and OpenBMB's community make it especially relevant to builders following Tsinghua-lineage open-source work.
A new paper from Tsinghua University, "Competing Learning in Hybrid Architectures", finds that the layer type trained first in hybrid attention-RNN models determines which mechanism masters retrieval. In the default joint setup, recurrent layers learn retrieval early, starving softmax attention of learning signal. The team proposes BLD, a schedule that freezes recurrent blocks for the first roughly 10% of pretraining tokens and then unfreezes them for joint training.
Tests across six KDA hybrid models spanning 180M to 1B parameters used identical architecture, data and token budgets. At the 1B scale, BLD lifted RULER from 32.75 to 36.04, Recall from 36.50 to 38.14, and improved LongPPL from 7.69 to 6.50. The gains required no additional compute or data.
Hybrid architectures that mix attention layers with linear-recurrent layers are increasingly used to cut long-context serving costs. The paper suggests training-order design is an overlooked lever for such models, and the code and paper are publicly available for teams pretraining hybrids.
Hybrid architectures: attention and RNN layers compete to learn the same skill 📈
New paper “Competing Learning in Hybrid Architectures” from Tsinghua:
• Competing Learning: recurrence learns retrieval first, leaving less learning signal for softmax attention
• BLD: freeze recurrent blocks for the first ~10% of pretraining tokens, then unfreeze and train jointly
• Better RULER and recall across six KDA hybrids (180M–1B)
• At 1B: RULER 32.75 → 36.04; Recall 36.50 → 38.14; LongPPL 7.69 → 6.50
Same architecture, data and token budget. Who learns first matters.
📄 https://t.co/AqLx6ixZCv 💻 https://t.co/CZ2TTX7eoN
#LLM #LongContext #HybridArchitecture #Attention #AI
This report is categorized under AITOP's "Papers" desk, focusing on China's AI industry and domestic foundation model ecosystem. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include hybrid architecture, RNN, attention, long context.
Source attribution: curated from OpenBMB (@OpenBMB).