PORT：基于 token 级重加权的噪声偏好优化框架
PORT: Preference Optimization via Robust Token-Level Reweighting
训练时偏好数据常有标注噪声，这套方法能在 token 级别识别并压低被污染 token 的影响，而且只加一次前向传播，训练成本几乎不变。做 RLHF 或 DPO 微调的团队可以直接用他们开源的代码试试，和序列级方法比抗噪能力更强。
俄亥俄州立大学研究团队提出 PORT（Preference Optimization via Robust Token-level reweighting），用于在偏好标注存在噪声时改进 LLM 对齐效果。现有鲁棒偏好优化方法主要在序列层面操作，将整条回复视为一致正确或错误，但被拒绝的回复仍可能包含有价值的推理步骤，被偏好的回复也可能含有细微错误。
PORT 的核心机制是在 token 级别进行重加权：利用 token 对数概率的经验累积分布函数（CDF），构建一个只需前向传播的代理指标来近似梯度范数惩罚，从而选择性地抑制被噪声污染的 token，不增加反向传播的计算开销。理论上，作者证明 PORT 通过最小化噪声偏好标签引起的梯度偏差项的上界来降低梯度偏差。
在多种噪声设置下的实验中，PORT 的一致性和鲁棒性均优于现有序列级基线方法。代码已在 GitHub（osu-srml/PORT）开源，可复现其偏好对齐训练流程。
PORT: Preference Optimization via Robust Token-Level Reweighting
Preference optimization has become a central approach for aligning large language models with human values, but its effectiveness depends heavily on the quality of preference annotations. In practice, preference data is often noisy due to annotation errors and ambiguity. Existing robust preference optimization methods primarily operate at the sequence level, implicitly treating entire responses as uniformly correct or incorrect. However, rejected responses may still contain informative reasoning steps, while preferred responses can include subtle errors. In this work, we propose \textbf{P}reference \textbf{O}ptimization via \textbf{R}obust \textbf{T}oken-level reweighting \textbf{(PORT)}, a fine-grained framework for robust alignment under noisy preferences. PORT performs token-level reweighting using the empirical cumulative distribution function (CDF) of token log-probabilities, yielding an efficient forward-pass-only proxy to gradient-norm penalization that selectively suppresses corrupted tokens without additional backward-pass computation. We provide theoretical analysis showing that PORT reduces the gradient bias from noisy preference labels by minimizing an upper bound on the bias term. Extensive experiments across diverse noise settings demonstrate that PORT consistently improves robustness and outperforms existing sequence-level baselines. The code is available at https://github.com/osu-srml/PORT
This report is categorized under AITOP's "Papers" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows and US-China foundation model benchmarks and commercialization comparison. Key entities and technical topics under continuous tracking include 偏好优化, PORT, RLHF, DPO.
Source attribution: curated from arXiv cs.AI (@Ding Zhu, Xiukun Wei, Tian Xie, Zhihui Zhu, Xueru Zhang, Mohammad Mahdi Khalili).