这篇论文教你用自然语言批评代替标量信号，从糟糕的演示中学更好，LC-BC和LC-DP在多个任务上都赢了。
提出语言批评模仿学习框架，利用自然语言作为结构化监督信号替代标量反馈。通过构建演示的语言标签，明确描述进度、识别次优行为并提供纠正指导。引入语言批评损失，实例化为LC-BC（行为克隆）和LC-DP（扩散策略）两种变体。理论证明该目标上界专家性能差距。在导航、操作和游戏等连续控制任务上，LC-BC和LC-DP优于强模仿学习和离线强化学习基线。
Language-Critique Imitation Learning from Suboptimal Demonstrations
Prior work on imitation learning from suboptimal demonstrations typically relies on compressed supervision signals such as confidence estimates, discriminator scores, or importance weights. These scalar signals are inherently limited, as they cannot explicitly express intermediate reasoning about task progress, failure modes, or corrective actions. We propose a language-critique framework for imitation learning from suboptimal demonstrations that instead leverages natural language as a structured supervision signal, avoiding the collapse of expressive feedback into scalars. Our method first constructs language labels from demonstrations that explicitly describe current progress, identify suboptimal behaviors, and provide fine-grained corrective guidance. We then introduce a language-critique loss that directly trains policies using these structured signals without reducing them to scalars, and instantiate it for both behavior cloning and diffusion policies, yielding LC-BC and LC-DP. We further provide a theoretical result showing that the proposed objective upper-bounds the expert performance gap under standard assumptions. Empirically, we evaluate on diverse continuous control tasks spanning navigation, manipulation, and gameplay, where our methods consistently outperform strong imitation learning and offline reinforcement learning baselines. These results demonstrate that language can serve as a powerful and structured form of supervision for learning robust policies from suboptimal data.