PLC-DPO Posterior Label Correction Method
PLC-DPO: Posterior Label Correction in Noisy and Ambiguous Preference Optimization
PLC-DPO outperforms existing preference optimization methods, offering researchers a more robust approach for handling real-world noisy preference data.
PLC-DPO improves Direct Preference Optimization by handling noisy and ambiguous preference labels. The method achieves a 60.5% mean win rate against DPO across 57 dataset-model-benchmark cells, compared to 55.5% for the next-best method. It uses calibrated policy-reference margin as online evidence to route training signals as clean, flip, or tie cases. The approach remains stable under injected-noise and tie stress tests, distinguishing flipped from weakly directional pairs.
PLC-DPO: Posterior Label Correction in Noisy and Ambiguous Preference Optimization
Direct Preference Optimization (DPO) simplifies alignment through pairwise comparisons but assumes all observed preferences are reliable. Real data often violates this assumption, leading to reversed, weak, or ambiguous labels that cause harmful policy updates. To address this, we propose Posterior Label Correction DPO (PLC-DPO) to robustly optimize preferences by routing each pair's training signal as a clean, flip, or tie case. The key idea is to use the calibrated policy-reference margin as online evidence to take appropriate correction actions. This reframes noisy preference learning as actively correcting supervision direction and strength rather than merely filtering suspicious examples. Across 57 dataset-model-benchmark cells, PLC-DPO obtains the best mean win rate against DPO (60.5 vs. 55.5 for the next-best method). Injected-noise and tie stress tests, human disagreement analysis, and self-confirmation diagnostics further show that the routing remains stable and distinguishes flipped from weakly directional pairs.
