PapersSelected9月15日 16:59

Systematic RLHF book covers DPO, RLVR and model personality shaping

shao__meng@shao__meng

Post

Why we picked it

For anyone working on or studying reinforcement learning from human feedback, this book provides a comprehensive guide covering key techniques like DPO and RLVR.

Nathan Lambert, former AI2 fine-tuning lead, has published the first systematic book on RLHF.

Systematic RLHF book covers DPO, RLVR and model personality shaping — Image · shao__meng

Body

The book follows the "instruction tuning → reward model → reinforcement learning" pipeline and covers topics like DPO, RLVR, synthetic data, over-optimization, and model personality shaping.

The full text is available online and the project code is open source.

Original · shao__meng

RLHF 首部系统专著：沿“指令微调 → 奖励模型 → 强化学习”主线，覆盖 DPO、RLVR、合成数据、过度优化与模型性格塑造。作者 Nathan Lambert（前 Ai2 后训练负责人），核心论点：数据远比算法重要。全书可在线阅读、项目代码开源。 x.com/i/article/2099…

Industry Coordinates & Source Verification

This report is categorized under AITOP's "Papers" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include RLHF, reinforcement learning from human feedback, DPO, model personality.

Source attribution: curated from shao__meng (@shao__meng).

#RLHF #reinforcement learning from human feedback #DPO #model personality #AI training

Keep reading

Recent stories with the same tag or section

How-to BuildersSelected00:04

Auto-Tuner Cuts RL Training Time Up to 4.93x

The non-obvious point is that most of the speedup came from configuration, not new hardware — teams may already have equivalent capacity locked behind untuned settings. Practitioners running RL fine-tuning pipelines, especially with DeepSeek models, should benchmark their own step times against these figures before provisioning more GPUs. The 40-GPU versus 56-GPU result is a direct cost argument for adding auto-tuning to training stacks.

#reinforcement learning #auto-tuning #DeepSeek #fine-tuning

Ate-a-Pi@svpinoSource

Papers Builders15:19

Chinese content研究：可验证 RL 训练中的奖励捷径与合规性学习

这篇论文讲清了 RL 训练里一个很实际的问题：25% 的冲突样本就能让模型学到 97.9% 的可盈利违规。作者还给出一个简单的二元惩罚方案，让安全完成率从 53.1% 提到 89.1%。做智能体训练和 RL 安全的工程师应该认真读。

#强化学习 #奖励破解 #智能体 #AI安全

arXiv cs.AI@Xinjie ShenSource

Papers Builders11:30

Chinese contentBRAT 训练方法解决多轮复用数据后训练崩溃问题

做 RL 后训练的人可以看看这篇：作者先解释了为什么同一批 rollout 多轮复用容易训崩，再给出 BRAT 这套更新方式，四次复用下不掉精度、种子间波动更小，比 PPO clipping 温和。

#BRAT #PPO #RLHF #强化学习

arXiv cs.LG@Xunlan Zhou et al. (7)Source

Industry BuildersMulti-source01:29

Chinese contentHandshake 一年做到 10 亿美元估值：给前沿实验室卖专业数据的生意

Latent Space 第一次拆数据实验室，主角是 Handshake 的 Garrett Lord。他们一年从零做到 10 亿美元估值，靠的是给 OpenAI、Anthropic 这类实验室供金融、生物、对齐领域的专家数据。做数据服务或评测创业的人能从中学到定价和交付的真实玩法。

#Handshake #OpenAI #Anthropic #数据标注

Event

Official accountLatent Space (swyx)@swyx .aiSource

How-to BuildersSelected01:03

Nathan Lambert Adds Score-Centering to the RLHF Book

Score-centering moving from a niche trick to an industry default signals how RL post-training recipes are converging, and the book chapter offers a derivation that clarifies why training-inference mismatch hurts policy optimisation. Practitioners implementing GRPO-style or StableReinforce-style pipelines can use it to stabilise advantage estimates. It pairs well with the book's existing chapters on reward modelling and policy gradients.

#RLHF #score-centering #StableReinforce #Tinker

Nathan Lambert@natolambertSource

Industry Builders01:03

Chinese contentNathan Lambert：post-training 与 post-post-training 是两种不同的技艺

RLHF 领域的研究员 Nathan Lambert 区分了 post-training 和 post-post-training，做模型训练的可以看他怎么说。

#Nathan Lambert #post-training #RLHF #开源模型

Nathan Lambert@natolambertSource