基于无渲染教师的端到端驾驶策略在线策略微调研究
OPTED: On-Policy Fine-Tuning for End-to-End Driving using a Render-Free Teacher
这篇论文讲的是一种叫OPTED的新方法，能更高效地训练自动驾驶模型，让模型在模拟环境中学习得更快，效果更好。
论文提出OPTED方法，通过一个使用强化学习训练的特权教师（基于HD-map和边界框）为预训练的学生模型提供监督，在闭环后训练阶段进行微调。该方法应用于TransFuser和VaVAM两个基于摄像头的模型，在AlpaSim中使用3DGS重建的真实驾驶日志进行微调，使驾驶分数分别提升1.6倍和9.5倍。在受控实验中，OPTED相比直接强化学习后训练，在保持接近人类先验的同时，减少了约三个数量级的模拟器交互次数。
OPTED: On-Policy Fine-Tuning for End-to-End Driving using a Render-Free Teacher
As scaling pre-training data alone yields diminishing returns, post-training is becoming increasingly important across physical AI domains such as autonomous driving. End-to-end driving policies are pre-trained in open loop with behavior cloning on human demonstrations. However, compounding errors during closed-loop deployment can take the vehicle outside the training data distribution, increasing the risk of safety-critical incidents. Closed-loop post-training can mitigate this risk but requires costly simulation for sensor-based policies. We propose OPTED (on-policy fine-tuning for end-to-end driving) which decouples reinforcement learning from the post-training of the end-to-end policy: a privileged teacher is trained using RL on vectorized inputs (HD-map and bounding boxes). This teacher then provides supervision to the pre-trained student during closed-loop post-training. We apply OPTED to two camera-based models, TransFuser and VaVAM, and fine-tune them in AlpaSim, using neural reconstructions (3DGS) of real driving logs. Driving scores increase by factors of 1.6$\times$ and 9.5$\times$, respectively. In controlled experiments OPTED matches closed-loop performance with approximately three orders of magnitude fewer simulator interactions than direct RL post-training, while staying closer to the human prior. Project page: https://01dami23.github.io/opted/
