Auxiliary Views Improve LLM Knowledge Acquisition
Knowledge Acquisition During Pre-training? Large Language Models Learn Better With Auxiliary Views
The findings offer practical insights for optimizing LLM training data efficiency, potentially reducing computational costs while improving knowledge retention.
Researchers conducted controlled experiments on how large language models acquire knowledge during pre-training. They found that allocating tokens from document repetition to auxiliary views improves learning, even for factual recall. The effectiveness of these auxiliary views does not depend on the strength of the teacher model that generates them. The study identified contextual and foundational forms of knowledge that aid learning when prior knowledge gaps exist.
Knowledge Acquisition During Pre-training? Large Language Models Learn Better With Auxiliary Views
Gaps remain in our understanding of how large language models (LLMs) acquire knowledge during pre-training. We posit that auxiliary views, reformulations of knowledge, are causally helpful for learning. We design controlled experiments to isolate this. First, we confirm that repetition is necessary for acquisition and clarify that paraphrasing helps only at smaller batch sizes. Second, holding the token budget fixed, allocating tokens from document repetition to auxiliary views improves learning, counterintuitively, even for factual recall. Third, the effectiveness of auxiliary views is not contingent on the strength of the teacher model that generates them. Fourth, we identify forms of knowledge, contextual and foundational, that aid learning in the presence of prior knowledge gaps. Finally, we examine how these effects manifest mechanistically via layer-wise biases and compression. Together, our findings suggest that auxiliary representations of knowledge, which arise naturally in large pre-training corpora, are a key factor in the success of pre-training and offer a plausible explanation for why data diversity matters.
