这篇论文测试了ChatGPT、Claude、DeepSeek写科研计划的能力，发现人类评委分不出好坏，但AI评委偏爱AI写的。值得一读。
该研究测试了大型语言模型（LLMs）在科研项目规划和提案评估中的表现。人类研究者与ChatGPT、Claude、DeepSeek（2025年中模型）分别生成了8个专家构思的研究项目的一页计划，共计32份提案。4位人类评审员和2个前沿LLM（Claude Opus 4.8、ChatGPT Pro 5.5）采用四维度评估标准盲审。人类评审员对AI和人类提案的整体评分相似，而AI评审员对AI提案的评分平均高出约1分（5分制）。人类评审员正确识别人类和AI提案的比例分别为72%和79%，而两个AI评审员均100%正确分类。研究表明当前LLM能生成与人类提案相当的计划，但AI评审员存在系统性偏好。
AI's Capability in Assisting Scientific Research in Physics, Astrophysics, and Cosmology II: Project Planning and Proposal Evaluation
We investigate how well large language models (LLMs) can assist scientific project planning and proposal evaluation. One-page project plans were independently generated for eight expert-conceived research projects in physics, astrophysics, and cosmology by human researchers and three contemporary LLMs (ChatGPT, Claude, and DeepSeek; mid-2025 models, used with their default tool access). The resulting 32 proposals were blindly evaluated by four human reviewers and two newer frontier LLMs (Claude Opus 4.8 and ChatGPT Pro 5.5) using a four-aspect evaluation rubric. Reviewers were also asked to identify whether each proposal was written by a human or an AI. Human reviewers rated human- and AI-written proposals similarly overall, whereas both AI reviewers scored AI-written proposals about one point higher (on a five-point scale) than human-written proposals. Human reviewers correctly identified human- and AI-written proposals 72% and 79% of the time, respectively, while both AI reviewers correctly classified all 32 proposals (100%). These results suggest that current LLMs can produce project plans comparable to human-written ones in the eyes of human reviewers, but that AI reviewers show a systematic preference for AI-generated proposals. Our results suggest caution when deploying LLMs widely in proposal preparation and evaluation.