SWHForest：为大规模代码语料库提供结构化筛选与检索层
SWHForest: A Structural Curation and Search Layer for Large-Scale Code Corpora
做代码模型预训练的同学都懂，想从几 TB 语料里挑干净子集有多痛苦。SWHForest 只存 AST 结构摘要不存源码，先用 SQL 筛再下载，实测砍掉 46-72% 下载量，把小时级检索压到秒级。训练数据团队和语料库维护方都该看看。
arXiv 论文 SWHForest 提出一个开源框架和公开数据集，为基于文件指针发布的代码语料库补上缺失的结构化筛选层。它对存档做一次遍历，把每个文件解析成 AST 后立即丢弃源码，只保留结构摘要，因此可以在不重新分发源代码的前提下提供服务。
在实际使用中，研究者可以用 SQL 或索引交集先筛出匹配文件，再只下载幸存的候选者，典型场景包括挑选文档完善的文件、过滤重复模板代码、审计基准污染或检索特定代码模式。在 Stack-Edu 十大语言的 5210 万个唯一文件和 3.41 亿个定义上，SWHForest 能在毫秒级响应筛选查询，下载量降低 46-72%。
对预训练代码模型的团队来说，这意味着把过去需要数小时逐语言流式下载解析的结构检索压缩到秒级或分钟级，省去了为了筛选子集而下载并解析数百万文件的前置成本。
SWHForest: A Structural Curation and Search Layer for Large-Scale Code Corpora
Pre-training code language models relies on massive code corpora usually streamed over the internet. However, researchers rarely need a multi-terabyte archive in its entirety; instead, their workflow centers on curated subsets: selecting well-documented files, filtering out repetitive boilerplate, auditing for benchmark contamination, or retrieving specific code patterns. Yet, because modern corpora are typically published as file pointers and basic attributes rather than source text, no general structural layer exists to select which files to download: curating a subset requires downloading and parsing millions of files first just to discard most of them. We present SWHForest, an open-source framework and public dataset that provides this missing structural layer for any pointer-based corpus without redistributing source code. A single pass over the archive parses each file into an AST and immediately discards the source text, keeping only its structural summary. Queried locally or exposed by archive providers as a server-side filtering API, SWHForest selects matching files first (via SQL or index intersection) and downloads only surviving candidates. Across the ten largest languages of Stack-Edu (52.1M unique files, 341M definitions), SWHForest answers curation queries in milliseconds while cutting downloads by 46-72%, reducing live structural searches from hours of per-language streaming to seconds or minutes.
This report is categorized under AITOP's "Papers" desk. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include SWHForest, Stack-Edu, AST, 代码语料库.
Source attribution: curated from arXiv cs.LG (@Federico Pennino, Andrea Gurioli, Maurizio Gabbrielli, Paolo Ferragina).