OctoLong 用跨仓库代码上下文训练长上下文模型，替换12%数据就在检索和代码理解上更强，模型已开源。
OctoLong 是一条上下文工程流水线，用 AST 解析器、语言服务器后端和包管理器递归检索代码引用，构建数百万 token 的跨仓库代码上下文。研究团队基于 600M 到 14B 参数的基座模型，用约 500 亿 token 混合数据（其中 OctoLong 代码上下文占 62 亿）进行中段训练，随后再做 100 亿 token 指令微调，得到 OctoLong-Instruct 系列。与 18 个开源长上下文模型对比，只需将传统扩展语料替换 12% 为 OctoLong 数据，就在长距离检索、长期状态跟踪、仓库级理解和智能体任务上显著提升。短上下文编码场景下的 API 调用能力也得到增强。
OctoLong: Mid-Training On Cross-Repository Code Contexts Enhances Long-Context Modeling
Context lengths of language models (LMs) have dramatically increased, driven by the demands for in-context learning, self-improvement, and long-horizon agentic workflows. Existing long-context corpora, however, are dominated by books, academic articles, and code repositories, which are finite resources and often scarce in long-distance dependencies. In this work, we introduce OctoLong, a context engineering pipeline that instruments an AST parser, a language server backend, and a package manager to facilitate the recursive retrieval of code references, enabling the curation of dependency-rich code contexts of millions of tokens in length. We then train OctoLong-Instruct, a suite of capable long-context open LMs, derived from base models ranging in size from 600M to 14B parameters, via context-extension mid-training on a ~50B-token mixture containing ~6.2B tokens of OctoLong code contexts, followed by ~10B tokens of instruction tuning. Our training ablations and experimental evaluations against 18 state-of-the-art open-weight long-context LMs show that supplanting just 12% of traditional context-extension corpora with OctoLong data yields substantial gains in long-range retrieval, long-term state tracking, repository-level code understanding, and downstream agentic tasks, while also enhancing API usage in short-context coding scenarios.