AutoScaffold：用冻结教师模型自动生成 RL 训练脚手架
AutoScaffold: AI-Guided Scaffolding for Reinforcement Learning of Language Models
训练 RL 模型时奖励只有一个数字，浪费了大量信息。这篇让教师模型自动写提示和策略当脚手架，四类任务全赢过普通 RL 基线，数学只用三分之一步数就超过 QuestA。做后训练的工程团队可以直接参考这套自动化方案。
arXiv 论文 AutoScaffold 提出一个 AI-for-AI 框架：RLVR 训练中，冻结的教师模型定期分析学生的训练轨迹，自动设计两类脚手架——跨任务共享的策略和针对个别难题的提示，并在学生进步后动态调整或移除。
机制上，教师诊断每个任务类的具体失败原因，脚手架随学习进度从分步流程转向针对性修补；评测时脚手架全部移除，只考察学生自身能力。
实验覆盖 ALFWorld、搜索式 QA、数学推理和 GPU kernel 生成四个领域：ALFWorld 成功率提升 7.2 分，QA 精确匹配平均提升 4.0 分，GPU kernel 超过 PyTorch 参考实现的正确任务比例提升最多 5.0 分；数学上学生以约三分之一的训练步数超过 QuestA 报告的准确率。
AutoScaffold: AI-Guided Scaffolding for Reinforcement Learning of Language Models
Reinforcement learning with verifiable rewards (RLVR) produces rich training experience, yet outcome rewards reduce each attempt to a single number and leave most of this information unused. Training-time scaffolds, such as strategies or hints added to the prompt during training, offer a natural way to turn this information into guidance, but their design and adaptation are largely left to humans and tailored to specific domains. We introduce AutoScaffold, a unified AI-for-AI framework that turns a student's training experience into adaptive scaffolds, with minimal domain-specific information and no human intervention during training. A frozen teacher periodically examines this experience, diagnoses bottlenecks, and designs scaffolds in two forms: (i) strategies shared across a class of tasks and (ii) hints for individual hard tasks, adjusting the content and amount of support as the student learns. The scaffolds are removed at evaluation. We apply the same framework to ALFWorld, search-based QA, mathematical reasoning, and GPU kernel generation. When evaluated without scaffolds, students trained with AutoScaffold outperform those trained with the corresponding RL baselines in all four domains, with gains of 7.2 points in ALFWorld success and 4.0 points in average QA exact match. On GPU kernel generation, AutoScaffold increases the fraction of tasks with a correct kernel that runs faster than the PyTorch reference by up to 5.0 points. On math, our student surpasses QuestA's reported accuracy using approximately one-third as many student training steps. Analyses show that the teacher identifies a different failure in each task class and shifts its scaffolds from step-by-step workflows to targeted fixes as the student improves, and that its revised scaffolds help the student learn more than keeping the previous scaffold or using no scaffold.
This report is categorized under AITOP's "Papers" desk. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include RLVR, AutoScaffold, 强化学习, 推理模型.
Source attribution: curated from arXiv cs.LG (@Zijian Zhang, Rizhen Hu, Jiangweizhi Peng, Shiyang Li, Sujay Bhatt, Sihan Zeng, Mingyi Hong).
