NVIDIA adds 1/2 DPO training to NeMo Megatron Bridge
Offline reference scoring turns DPO from a two-model job into a one-model job, roughly halving memory and removing per-step reference overhead during preference tuning. Teams running preference optimisation on Megatron-scale models can reuse one scoring pass across full hyperparameter sweeps, cutting GPU hours. It matters most for builders fine-tuning large models where a second reference copy was previously the bottleneck.
NVIDIA's NeMo Megatron Bridge training library now supports a 1/2 DPO implementation that runs on a single model per GPU, matching SFT in memory usage and step time. Standard DPO keeps a frozen reference model copy that is re-run at every step, even though its logprobs never change. The new implementation scores the reference model once offline and saves chosen/rejected logprob sums per pair to disk, which training then reads back. The stored file contains raw sums without beta applied, so a single scoring run supports every subsequent hyperparameter sweep with different beta and learning rate values.
1/2 DPO in NeMo™ Megatron Bridge, @NVIDIA’s training library for Megatron-based models, now trains like SFT: one model on the GPUs, same memory, same step time.
DPO's reference model is frozen, so its logprobs over your dataset never change. Yet the usual setup keeps a second copy of the model and runs it at every step.
Our implementation scores the reference once (offline scoring) and saves chosen/rejected logprob sums per pair to disk. Training reads them back instead.
The best part: the file stores raw sums, with no beta applied. Beta and learning rate only enter at loss time, so one scoring run covers every hyperparameter sweep you do afterwards. >>
