这篇论文提出FORE方法，只用占用率可实性就能做离线策略评估，省去Bellman完备性的复杂条件，适合研究强化学习的同学参考。
FORE通过伴随Bellman递归直接估计折扣占用率，仅需占用率可实性条件，无需Bellman完备或投影算子稳定性。在总体层面，KL投影递归以相对熵形式向真实比率收缩。经验递归的有限样本遗憾界表明，KL收敛误差由对数比率逼近误差和假设类复杂度决定的统计误差组成。该方法支持奖励加权估计、占用加权拟合Q评估以及双重稳健估计。这一结果将折扣占用率可实性识别为离线策略评估的充分条件。
Fitted Occupancy-Ratio Evaluation without Bellman Completeness
Occupancy ratios correct distribution shift in offline reinforcement learning and are central to off-policy evaluation. Existing primal-dual and minimax methods typically estimate these ratios by enforcing occupancy-balance moments over a critic class. We propose fitted occupancy-ratio evaluation (FORE), a fitted fixed-point method that characterizes the discounted occupancy ratio through an adjoint Bellman recursion. At each iteration, FORE solves a single-level density-ratio objective on one-step-transition data, thereby projecting the adjoint Bellman image onto a log-ratio class in Kullback--Leibler (KL) divergence. Unlike analyses of fitted Q-evaluation, which typically require value-function realizability together with Bellman completeness or projected-operator stability, our central approximation condition is just realizability of the discounted occupancy ratio itself. Under this condition, the population KL-projected recursion contracts in relative entropy toward the true ratio by virtue of the adjoint Bellman operator being a KL-contraction. For the empirical recursion, we establish finite-sample regret bounds that yield convergence in KL up to log-ratio approximation error and a statistical error governed by the complexity of the ratio hypothesis class. The fitted ratio supports direct value estimation by reward reweighting, occupancy-weighted fitted Q-evaluation, and doubly robust estimation that combines the fitted ratio with a fitted Q-function. Together, these results identify discounted occupancy-ratio realizability as a sufficient condition for offline policy evaluation without any completeness assumptions.