想搞懂模型知识边界怎么控制？这篇论文用小学课程语料训练了个5B模型，还公开了数据和模型，适合研究预训练数据影响的人看。
研究者推出LITTLECURRICULUM，一个包含880亿token的预训练语料库，内容限定为美国小学五年级以下的知识范围。他们用该语料从头训练出5B参数的LittleLearner模型，模型语言能力足以进行开放式评测，但知识边界清晰对应课程大纲。实验表明，通过后训练和上下文学习注入新知识，LittleLearner能更好利用已有知识，但无法提升超出课程范围的能力。该研究为可控条件下研究模型知识获取提供了沙盒环境。
LittleLearner: Language Models Under Pedagogically Controlled Knowledge Exposure
Modern language models are trained on heterogeneous web-scale text corpora. Consequently, studying knowledge and skill acquisition is difficult, as prior exposure to related content is hard to characterize. To address this challenge, we introduce LITTLECURRICULUM, a curated 88B-token pretraining corpus tailored to U.S. elementary school material, explicitly excluding concepts, facts, and vocabulary taught above Grade 5. Training a 5B-parameter LLM from scratch on LITTLECURRICULUM yields LITTLELEARNER, a model with sufficient language competence for open-ended evaluation, yet with clear knowledge and capability boundaries mapped to interpretable curriculum guidelines. We release LITTLECURRICULUM and LITTLELEARNER as a developmentally restricted sandbox to study how models acquire, represent, and use data under a well-defined training scope. We illustrate the sandbox's utility in a first suite of experiments on injecting new knowledge through post-training and in-context learning. These methods let LITTLELEARNER better utilize existing knowledge, but do not raise out-of-scope capabilities. Our findings underscore the value of this controlled environment for future investigations.