想解决机器人奖励函数跨环境失效的问题？这篇论文给出了数学分析和一种更聪明的教学策略，用更少反馈让奖励更鲁棒。
该论文研究如何在多环境下学习鲁棒的奖励函数，避免逆强化学习（IRL）因单环境演示而过拟合。作者分析了不同反馈模态（如比较、演示）对奖励约束的强度，发现比较模态在无限数据下提供更强的全局约束。提出一种分层机器教学算法，先贪婪选择互补环境，再在最优环境下查询低成本反馈。在实验中，该方法在相同反馈预算下相比均匀基线显著降低了后悔值（regret），并提升了到未见过环境的泛化能力。
Multi-Modal, Multi-Environment Machine Teaching for Robust Reward Learning
As autonomous agents are increasingly deployed across diverse operational contexts, aligning their behavior with human intent demands reward functions that remain robust to such changes rather than overfitting to any single environment. Inverse reinforcement learning (IRL) provides a principled way to infer such objectives from human feedback. However, existing analyses of optimal teaching approaches for IRL focus on single-environment, demonstration-only settings, leaving underexplored how heterogeneous feedback modalities and environment dynamics jointly constrain reward functions that generalize across multiple environments. Because demonstrations in one MDP entangle reward information with that environments specific structure, the resulting rewards frequently fail to generalize when the agent is deployed in a new setting. We first analyze how different feedback modalities constrain rewards, showing that, in the unlimited-data regime, comparisons impose strictly stronger global constraints than other modalities. Beyond this theoretical analysis, we introduce a hierarchical machine teaching algorithm for reward learning that operates across multiple MDPs. The algorithm first greedily selects informative environments that expose complementary reward constraints, then strategically queries low-cost feedback within those environments. Empirically, our method achieves substantially lower regret and stronger generalization to held-out environments than uniform teaching baselines under identical feedback budgets, demonstrating the importance of multi-environment, multi-modal teaching for learning dynamics-robust reward functions.