LittleLearner：在小学课程知识范围内训练的5B参数语言模型
LittleLearner: Language Models Under Pedagogically Controlled Knowledge Exposure
想搞懂模型知识边界怎么控制？这篇论文用小学课程语料训练了个5B模型，还公开了数据和模型，适合研究预训练数据影响的人看。
研究者推出LITTLECURRICULUM，一个包含880亿token的预训练语料库，内容限定为美国小学五年级以下的知识范围。
他们用该语料从头训练出5B参数的LittleLearner模型，模型语言能力足以进行开放式评测，但知识边界清晰对应课程大纲。实验表明，通过后训练和上下文学习注入新知识，LittleLearner能更好利用已有知识，但无法提升超出课程范围的能力。
该研究为可控条件下研究模型知识获取提供了沙盒环境。
LittleLearner: Language Models Under Pedagogically Controlled Knowledge Exposure
Modern language models are trained on heterogeneous web-scale text corpora. Consequently, studying knowledge and skill acquisition is difficult, as prior exposure to related content is hard to characterize. To address this challenge, we introduce LITTLECURRICULUM, a curated 88B-token pretraining corpus tailored to U.S. elementary school material, explicitly excluding concepts, facts, and vocabulary taught above Grade 5. Training a 5B-parameter LLM from scratch on LITTLECURRICULUM yields LITTLELEARNER, a model with sufficient language competence for open-ended evaluation, yet with clear knowledge and capability boundaries mapped to interpretable curriculum guidelines. We release LITTLECURRICULUM and LITTLELEARNER as a developmentally restricted sandbox to study how models acquire, represent, and use data under a well-defined training scope. We illustrate the sandbox's utility in a first suite of experiments on injecting new knowledge through post-training and in-context learning. These methods let LITTLELEARNER better utilize existing knowledge, but do not raise out-of-scope capabilities. Our findings underscore the value of this controlled environment for future investigations.
This report is categorized under AITOP's "Papers" desk. Key entities and technical topics under continuous tracking include LittleLearner, LITTLECURRICULUM, pretraining, knowledge boundaries.
Source attribution: curated from arXiv cs.LG (@Fanfei Li, Jana Zeller, Manuel Prada-Corral, Thaddäus Wiedemer, Prasanna Mayilvahanan, Ryan Cotterell, Wieland Brendel).
