想处理真实场景奖励缺失的强化学习玩家可以看这篇，用影子变量和桥函数解决偏差问题，实验比传统方法稳。
离线强化学习中，即时奖励常因记录稀疏或审查而缺失，导致评估偏差。本文聚焦奖励缺失非随机(MNAR)场景，在有限时域MDP下研究离线策略评估(OPE)。作者利用未来状态作为影子变量，结合奖励依赖倾向模型辨识全数据条件均值奖励。进一步引入桥函数并通过min-max估计避免双重采样，提出Fitted-Q-Evaluation风格估计器。在模拟数据和MIMIC-III Sepsis数据上，该方法在误差和一致性上优于现有基线。
Off-Policy Evaluation for Missingness-Aware Policies in MDPs with Rewards Missing Not at Random
In offline Reinforcement Learning, immediate rewards in logged batch data are often unobserved due to sparse or irregular record-keeping, or censored beyond certain reward values. This issue arises in practical settings, including health care and marketing. We investigate off-policy evaluation (OPE) in finite-horizon Markov decision processes when rewards are missing not at random (MNAR), which breaks ignorability and induces selection bias even after conditioning on states and actions. To address this, we formalize a reward-dependent propensity model and use future states as shadow variables to identify the full-data conditional mean reward. We further introduce a bridge function that recovers the conditional mean reward without explicitly modeling the MNAR mechanism, and estimate it via a min-max procedure to avoid double sampling. Building upon these identification results, we propose an Fitted-Q-Evaluation-style estimator that propagates the recovered rewards while allowing target policies to depend on past missingness indicators. Finally, we establish consistency and finite-sample error bounds for our OPE estimator, and show through experiments the strong performance of our method compared to existing methods on simulated and MIMIC-III Sepsis data.