DNO 方法让 Llama 3.1 8B 在 AlpacaEval 2.0 上胜率升至 45.3%
Aligning Language Models with General Preferences: An Empirical Study of Direct Nash Optimization
一篇对齐方法的复现研究：用纳什均衡思路微调，8B 模型干过了 70B，还验证了不用 GPT-4o 标注也行，做对齐的可以看看。
论文复现了 Direct Nash Optimization（DNO-Prct），通过近似双人偏好博弈的纳什均衡来对齐语言模型。在 Llama 3.1 8B Instruct 上，三轮迭代将 AlpacaEval 2.0 的 LC 胜率从 31.7% 提升到 45.3%，Mistral 7B 从 6.8% 升至 18.4%，调优后的 8B 模型各项胜率超过 Llama 3.1 70B Instruct。用同家族开源模型替代 GPT-4o 做标注时，仍保留 62%（Llama）和 82%（Mistral）的 LC 增益。消融实验显示去掉 gold 回复或数据回看会损失 6 到 10 个 LC 点。作者同时指出所有结果为单次运行，且 GPT-4o 同时担任教师与评测裁判。
Aligning Language Models with General Preferences: An Empirical Study of Direct Nash Optimization
Direct Nash Optimization (DNO) aligns a language model to a general preference function by approximating the Nash equilibrium of a two-player preference game. Its practical variant, DNO-Prct, iterates on-policy sampling, LLM-based rating of the samples together with a teacher's gold response, large-margin preference-pair selection and a DPO-style contrastive update. We re-implement DNO-Prct independently and study it on two open-weight instruct models, Llama 3.1 8B Instruct and Mistral 7B Instruct v0.1, with GPT-4o or a larger open-weight model of the same family as the annotator. On AlpacaEval 2.0 with GPT-4o as judge, three iterations raise the length-controlled (LC) win rate against GPT-4o from 31.7% to 45.3% for Llama and from 6.8% to 18.4% for Mistral, and the tuned 8B model scores above Llama 3.1 70B Instruct on every reported win rate. With same-family open-weight annotators, the runs retain 62% (Llama) and 82% (Mistral) of the LC gain achieved with GPT-4o as annotator, so the rating step does not require a proprietary model. In a separate ablation configuration, variants that remove the gold responses, remove the data look-back, or accept only teacher-preferred pairs score 6 to 10 LC points below the full procedure, whereas varying the number of on-policy samples per prompt (1 to 7) moves the LC win rate by at most 2.1 points, and a margin sweep (1 to 5) run in its own pipeline by at most 2.4 points. We also report the cost of the pipeline and where the practical algorithm departs from the theory it approximates. All results are single runs, and GPT-4o is both the teacher and the evaluation judge; we discuss what this implies for the reported gains.