CO-LMLM用360M参数打平gpt-4o-mini，事实准确度比Claude还高，这是用更小模型做更好知识检索的路线。
CO-LMLM将知识库中的连续键与文本知识值配对，摆脱了传统关系型知识库的查询限制。在Wikipedia和FineWeb-Edu上预训练，360M参数模型困惑度低于使用40倍数据训练的LLM。SimpleQA验证显示其性能与gpt-4o-mini相当，且高于Claude Sonnet 4.5。该架构在多个规模下均优于先前LMLM和普通LLM。
Co-LMLM: Continuous-Query Limited Memory Language Models
Limited memory language models (LMLMs) externalize factual knowledge during pretraining to a knowledge base (KB), rather than memorizing it in their weights. During generation, the model then fetches knowledge from the KB as needed. This recently introduced paradigm provides multiple advantages, including knowledge control capabilities that remain beyond conventional LLMs. We propose continuous-query LMLM (CO-LMLM), where the KB pairs continuous keys with textual knowledge values, a significant departure from prior reliance on relational KB and queries. CO-LMLM generates flexible vector queries at minimal cost, while still integrating human-readable and attributable retrieved knowledge into its generation. We pair this design with an annotation pipeline that tags free-form factual spans in arbitrary text, removing prior work's restriction to Wikipedia. Across pretraining on Wikipedia and FineWeb-Edu and at multiple model scales, CO-LMLM outperforms prior LMLMs and vanilla LLMs in both perplexity and factual precision. At 360M scale, this includes lower perplexity than models pretrained on 40x more data, and SimpleQA-verified performance that is in line with gpt-4o-mini and higher than Claude Sonnet 4.5.