这篇论文给科学问题生成系统立了个可证伪的考试：用未来文献打分。证据结构优先的生成器比纯 LLM 提示更靠谱，还发现 LLM 评委互相打分虚高，值得做科研工具的人看看。
arXiv 论文提出用历史回测评估科学问题生成系统：从历史截止点的语料生成问题，冻结后由未来语料判定其是否被回答、部分解决或独立提出。研究在 2010-2024 四个时间窗口测试 798 个问题，发现证据结构优先的生成器优于纯 LLM 提示，后者存在记忆性相关但缺乏前瞻性。七位评分者一致性研究显示人类标注者 kappa 仅 0.17，而前沿模型间一致性达 0.60，说明 LLM 评委可靠性被高估三倍。论文发布了可复现的天文学实例和 200 个冻结问题供未来验证。
Historical Backtesting for Scientific Question Discovery: A Protocol and Astronomy Pilot
Systems that generate scientific research questions are evaluated today by expert scores, LLM-as-judge ratings, or curated case studies -- all subjective, none falsifiable. We formalize historical backtesting as an alternative: a system generates questions from a corpus frozen at a historical cutoff, the questions are frozen before any access to later literature, and a temporally isolated future corpus then determines whether each question was subsequently answered, partially addressed, independently posed, or ignored, and whether its underlying premise was supported or refuted. The protocol is model-agnostic: any system that emits frozen questions can be scored. We release reproducible astronomy instances with temporally isolated corpora, frozen questions, auditable labels, four reference baselines, and a submission interface. Two findings result. First, evidence-structure-first generation outperforms LLM-only prompting: across a generator decomposition crossed with a four-cutoff stress test (2010-2024, 798 judged questions) whose last window postdates model training, LLM-only generation shows memorized relevance without specific foresight, while a generator using no model weights at all finds questions whose premises the future refutes in every era. Second, a seven-rater agreement study (two blinded human annotators, five judge models, 90 items) indicts the outcome taxonomy rather than the judge: two careful humans agree at kappa = 0.17, every judge model agrees with the professional annotator as well or better (0.17-0.26), and frontier models agree with one another at 0.60 -- certifying an LLM judge by model-model agreement would have overstated its reliability threefold. A prospective instance -- 200 questions frozen 2026-08-17, scored 2027-2030 -- is released so the central claims become contamination-free tests that time itself will grade.