做科学创新评估或使用 LLM 辅助审稿的团队，这篇论文揭示了 LLM 裁判的盲区——它可能高估新颖性，导致误判。建议点开了解 RQ-Bench 的测试方法，避免在关键评估中踩坑。
该研究指出，用 LLM 作为裁判来评估科学问题的新颖性存在严重缺陷。作者构建了 RQ-Bench 基准，基于 arXiv 论文提取作者锚定的研究问题，并与模型生成的问题进行对比。实验发现，LLM 裁判一致高估模型生成问题的新颖性，产生“新颖性幻象”，而领域专家则得出相反结论。此外，模型生成的问题往往狭窄或受限于源材料，LLM 裁判难以察觉。该结果对依赖 LLM 进行科学新颖性评估的可靠性提出严重质疑。
On the Limits of LLM-as-Judge for Scientific Novelty Assessment
LLMs are increasingly used to generate and judge scientific ideas. This makes novelty evaluation a central problem. Full idea evaluation is difficult because it often requires judging a method, its feasibility, and its empirical promise. We therefore study a cleaner upstream object: the research question (RQ). RQ generation is a prerequisite for scientific ideation, and RQs can be compared against questions pursued in real papers. We introduce RQ-Bench, a benchmark built from recent arXiv papers. For each paper, we reconstruct author-anchored RQs from its cited background, gaps, and contributions. These RQs are not the only valid questions for the same background. They are author-anchored reference points for testing novelty judgments. We evaluate model-generated RQs with standalone LLM judging, comparative LLM judging, and human expert evaluation. LLM judges consistently rate model-generated RQs as highly novel, producing a novelty mirage; in comparative evaluations, this preference becomes even stronger. Domain experts, however, reach the opposite conclusion and prefer the author-anchored reference questions. We further find that many generated RQs are narrow or source-bound, a dimension that LLM judges often miss unless explicitly tested. Overall, the contradictory novelty evaluations between LLM judges and human experts raise a serious concern about the reliability of using LLMs to assess the scientific novelty of research questions.