FireworksAI 的 RL 模型训练遇到数值问题
The hardest bugs were numerical. Long-horizon RL drifts when the engine generating rollouts and the ...
精选理由
FireworksAI 的 RL 模型训练遇到数值问题，调整分词对齐后解决了信号漂移问题。
该模型在训练过程中，生成和评估阶段对同一词元的概率分配不一致，导致长期规划时出现漂移。通过调整两者之间的分词对齐，确保了训练信号的可信度。
原文 · Fireworks AI
The hardest bugs were numerical. Long-horizon RL drifts when the engine generating rollouts and the ...
The hardest bugs were numerical. Long-horizon RL drifts when the engine generating rollouts and the one scoring them stop assigning the same probabilities to the same tokens. Aligning tokenization across both kept the training signal trustworthy. 💬 1 🔄 0 ❤️ 0 👀 130 📊 1 ⚡