这篇论文把奖励函数设计拆成三步，还有个只要 O(n log κ) 次偏好查询的算法，做 RLHF 或奖励建模的值得一读。
该论文提出一套形式化流程，让非专家也能构建符合人类偏好排序的奖励函数。流程分三步：先从自然语言任务中提炼基本目标并导出可测量的结果变量，再通过最小成本部分覆盖在因果DAG上选择奖励项，最后用偏好查询拟合权重。权重拟合被转化为凸可行性问题，借助分离预言机迭代收窄可行区域，只需 O(n log κ) 次偏好查询。这是首个保持确定性无冲突可行权重区域的奖励设计方法。
A Framework for Designing Reward Functions: From Objectives to Features to Human-Aligned Reward Functions
We present a formal process to enable non-experts to instantiate and iterate on human-aligned reward functions, i.e. reward functions that adhere to a given preference ordering over trajectories. Given a task described in natural language, our process produces a linear reward function in three steps: distill the task's objectives into a set of fundamental objectives and derive measurable outcome variables that capture those fundamental objectives, select a causally representative subset of outcome variables as the reward terms, and fit weights to those reward terms via preference elicitation. Our contributions describe the first step and formalize the latter two steps. The first is a guided workflow for deriving outcome variables. The second is a reduction of reward term selection to minimum-cost partial cover on a causal DAG, solved in polynomial time via max-flow. The third is a geometric framing of weight fitting as a convex feasibility problem iteratively narrowed by preference queries, solved by existing separation oracle methods. To the best of our knowledge, this is the first reward-design method that maintains a deterministically conflict-free feasible weight region, narrowed to a desired tolerance via a separation oracle with O(n log κ) preference queries.