这篇论文提出了LLM反馈的奖励塑造方法，即使在LLM评分不准确时也能保持最优策略，比传统方法更可靠。
该研究结合大型语言模型与强化学习，将LLM规划器和RL控制器架构形式化为目标增强马尔可夫决策过程。研究表明，当LLM的状态进展分数作为有界势函数时，即使LLM评分不准确，形成的奖励塑造项也能保持最优策略集。研究者在小型MDP上验证了结果，包括四种势能配置，其中一种对抗性配置放大到基础奖励幅度的20倍。
Policy-Invariant Reward Shaping from LLM Feedback: A Framework for Hybrid RL Agents
Combining large language models with reinforcement learning is increasingly explored, yet the theoretical status of LLM-derived reward signals is often left implicit. We formalize the hybrid LLM-planner and RL-controller architecture as a Goal-Augmented Markov Decision Process and show that when the LLM per-state progress score is used as a bounded potential function, the resulting shaping term preserves the optimal policy set even when the LLM scores are inaccurate. This guarantee is stronger than what general LLM-as-reward approaches provide. We verify the result numerically on a small MDP under four potential configurations, including an adversarial one scaled to twenty times the base reward magnitude.