偏好反馈是 RLHF 的核心，但理论分析一直稀缺。这篇论文把核 MDP 和偏好学习结合，给出了亚线性遗憾界，做理论强化学习或 RLHF 算法设计的研究者值得细读。
本文研究了在 episodic 核马尔可夫决策过程（MDP）中仅通过偏好反馈进行强化学习的问题。与传统的数值奖励不同，人类反馈通常以偏好形式出现（如比较两个轨迹的优劣），这更符合 RLHF 的实际场景。作者假设奖励和转移函数属于核函数空间（一种通用的理论分析模型），并设计了基于偏好的值估计和置信集方法，专门处理每轮结束时给出的二元偏好比较。理论结果表明，学习策略的遗憾值随回合数亚线性增长，即最终能收敛到最优策略。该工作为偏好反馈下的强化学习提供了严格的理论基础，尤其适用于奖励难以量化但人类容易比较的场景。
Learning Kernel-Based MDPs from Episodic Preferential Feedback
Human feedback often arrives as preferences rather than calibrated numeric rewards, motivating reinforcement learning from preferential feedback, also referred to as reinforcement learning from human feedback (RLHF). We present a rigorous theoretical study of preference-only learning in episodic kernel MDPs. In each episode, the learner deploys two policies from a common start state and receives a single binary label indicating which trajectory is preferred, modeled by a Bradley--Terry--Luce link on the difference of cumulative (unobserved) rewards. Under kernel-based assumptions on the reward and transition functions (one of the most general models amenable to theoretical analysis) we develop preference-based value estimation and confidence sets tailored to end-of-episode comparisons.We prove high-probability regret bounds that scale sublinearly in the number of episodes, implying that the value of the learned policy converges to that of the optimal policy.