研究人员发现小模型实验就能确定可转移的近最优区域，无需大规模搜索，大幅降低大模型训练成本。
研究解决了大模型后训练中监督微调(SFT)与强化学习(RL)的标注预算分配问题。该研究提出了近最优区域概念，即使在2-10%的小容忍度下，该区域也很宽。研究发现近最优区域随模型规模扩大而变宽，并能从小代理模型可靠转移到大型目标模型。研究结果在不同任务、模型家族和基于偏好的离策略与基于奖励监督的在策略RL方法中保持一致。
Scaling Near-Optimal SFT-RL Annotation Budget Allocation from Small to Large LLMs
How to divide a fixed annotation budget between supervised fine-tuning (SFT) and reinforcement learning (RL) during LLM post-training remains an open problem. Existing work characterizes only broad trends (e.g., SFT dominates in low-data regimes), lacks a principled allocation framework, and does not examine whether the optimal ratio transfers across model sizes. We frame this problem in terms of near-optimality: rather than seeking a single optimal SFT-RL ratio, we characterize the near-optimal region, the set of allocations within a specified tolerance of peak performance. Empirically, this region is wide even for small tolerances (2-10%), widens with model scale, and transfers reliably from small proxy models to large target models. This yields a practical strategy: small proxy-model experiments suffice to identify a transferable near-optimal region, eliminating the need for exhaustive large-scale search. Our results hold consistently across tasks, model families, and both preference-based off-policy and reward-supervision on-policy RL methods. We further analyze how the asymmetry in annotation costs between SFT and RL data shifts the near-optimal region.