想让机器人按你的偏好做事？FPL 把“快一点”“稳一点”这种主观要求变成可训练的奖励信号，效果比传统方法好 38%。
研究人员提出 Freeform Preference Learning (FPL)，让标注者通过自然语言定义速度、安全性等偏好轴，并沿各轴提供成对比较。FPL 学习语言条件奖励模型，在四个真实世界和两个模拟长程操作任务中，比稀疏奖励和二元偏好方法提升 38 个百分点。该方法无需显式子任务分割即可获得密集进度信号，且能组合未见过的行为，允许测试时无需重新训练即可引导策略偏向不同行为。论文附有博客与视频演示。
Freeform Preference Learning for Robotic Manipulation
Reward design remains a central bottleneck for autonomous robot policy improvement, especially in long-horizon manipulation tasks where sparse success labels provide too little signal and binary preferences collapse many competing notions of quality into one ambiguous signal. We introduce Freeform Preference Learning (FPL), a method for learning robot policies from freeform human preferences. Rather than asking annotators which of two trajectories is better overall, FPL lets them define natural-language preference axes, such as speed, safety, quality of placement, or carefulness, and provide pairwise preferences along each axis. These annotations are used to learn a language-conditioned reward model that maps a trajectory and preference label to an axis-specific reward. We use this model to train a reward-conditioned policy that optimizes across the multiple human-specified dimensions. Across four real-world and two simulated long-horizon manipulation tasks, FPL improves over sparse-reward and binary-preference methods by 38 percentage points. Beyond improved performance, FPL learns dense progress signals without explicit subtask segmentation, shows compositionality of behavior not present in the data, and allows users to steer the policy towards different behaviors at test time without retraining. Blog post with videos available at https://freeform-pl.github.io/fpl.website/