这篇论文用o4-mini试了不同提示词，发现它解题会翻车，还试了怎么教学生挑它毛病，搞物理教育的值得看看。
该研究以转动力学问题为测试，利用问题分类框架分析OpenAI o4-mini的解答，并用MAPS评分量表评估。结果显示，明确提示词能提升解答完整性，而欠指定和多模态提示暴露物理推理与正确性缺陷。24个物理实验小组参与评估，MAPS引导的批评比仅解题组更贴合专家视角，能识别未执行数值步骤和符号未定义等问题。研究为AI解题基准和物理教育中的批判性训练提供了实证基础。
Probing AI-generated physics solutions and preparing students to critique them
This study examines Artificial Intelligence (AI)-generated physics solutions from two connected perspectives: how prompt design shapes these solutions and how students can be prepared to critique them. Using a rotational-mechanics problem, we adapted a problem-classification framework to examine prompt variations, evaluating OpenAI's o4-mini responses with the Minnesota Assessment of Problem Solving (MAPS) rubric. Well-specified prompts improved solution completeness; underspecified and multimodal prompts exposed weaknesses in physics reasoning and correctness. In the student-evaluation phase, 24 introductory physics lab groups evaluated an o4-mini solution to this problem after either independently solving a related problem or critiquing its AI-generated solution with MAPS-based reflection questions. Problem-solving-only groups exhibited uncritical or misconception-based critiques; MAPS-guided groups identified more expert-aligned issues, including skipped numerical procedures and undefined notation. Together, our findings contribute to physics education research by showing how AI-generated solutions can ground both model-reasoning benchmarks and improved student critique of that reasoning through MAPS-based reflection.