做信息检索评估的团队终于有了低成本诊断工具——SPECTRA 用合成数据暴露系统瓶颈，比等人工标注快得多，做检索系统测试的开发者建议试试。
SPECTRA 是一个可复现的框架，用于生成合成文本语料库和检索测试集，通过分离潜在主题结构、表面文本实现、元数据控制、查询意图生成和确定性相关性预言，解决了人工标注成本高、文档隐私或设计阶段不可用的问题。该框架旨在作为 Cranfield 和 TREC 风格评估的诊断补充，而非替代。单进程 Python 原型生成了多达 6 万文档和 961 万 token 的语料库，保持可控的长尾词汇增长，并为 96 个查询生成分级相关性标签。实验显示，生成速度接近线性（每秒约 1.2 万至 1.4 万文档），估计的 Zipf 斜率绝对值接近 0.86，增加跨主题干扰文本使 BM25 nDCG@10 从 2% 干扰时的 1.00 降至 36% 干扰时的 0.43。这些结果表明，轻量级合成语料库可以在昂贵的人工构建之前暴露检索系统的扩展性和故障模式。
SPECTRA: Synthetic IR Test Collections with Relevance Oracles and Controlled Distractor Diagnostics
Scalable information retrieval testing needs corpora that are large enough to stress index construction, ranking latency, query routing, and evaluation tooling, yet human-judged test collections remain expensive and may be unavailable when documents are private or still under design. This paper introduces SPECTRA, a reproducible framework for generating synthetic text corpora and retrieval test collections through a separation of latent topical structure, surface text realization, metadata controls, query intent generation, and deterministic relevance oracles. The framework is intended as a diagnostic complement to Cranfield-style and TREC-style evaluation, not as a replacement for human assessment. A single-process Python prototype generated corpora up to 60,000 documents and 9.61 million tokens while preserving controllable long-tail vocabulary growth and producing graded relevance labels for 96 queries. In the local simulation study, generation remained close to linear at roughly 12K to 14K documents per second, estimated Zipf slopes stayed near 0.86 in absolute value, and increasing cross-topic distractor text reduced BM25 nDCG@10 from 1.00 at 2% distractors to 0.43 at 36% distractors. These results show that lightweight synthetic corpora can expose retrieval-system scaling and failure modes before costly collection construction begins.