这篇论文给IRL的超梯度计算找了个巧妙的捷径，用Fisher矩阵和谱草图省下大量算力，做强化学习的朋友可以看看。
逆强化学习（IRL）旨在从专家演示中恢复奖励函数，通常被建模为双层优化问题，内层为策略优化，外层衡量策略与专家数据的差异。然而，外层更新需要计算涉及内层目标逆Hessian向量积的超梯度，计算成本高昂。本文证明在内层最优处，内层目标的Hessian与策略的Fisher信息矩阵成正比，从而提出基于Fisher的超梯度，与自然超梯度下降紧密相关。为应对大规模Fisher矩阵的瓶颈，作者使用流式谱草图近似逆Fisher向量积，避免显式构造Fisher矩阵。在离散和连续控制环境中，该方法相比一阶随机双层基线，实现了有竞争力的策略性能和奖励排序质量，同时降低了曲率存储复杂度并提升了计算效率。
Efficient Hypergradient Descent for Inverse Reinforcement Learning
Inverse reinforcement learning (IRL) aims to recover a reward function under which the resulting policy reproduces the behavior observed in expert demonstrations. A natural approach is to formulate IRL as a bilevel optimization problem, in which the inner level corresponds to policy optimization under the learned reward and the outer level measures the discrepancy between the induced policy and expert data. However, this formulation is computationally challenging in practice because the outer update requires a hypergradient involving an inverse-Hessian-vector product for the inner objective. We address this challenge by showing that, at the inner optimum, the Hessian of the inner objective is proportional to the Fisher information matrix of the policy, yielding a structured Fisher-based hypergradient closely related to Natural Hypergradient Descent. To address the resulting scalability bottleneck associated with large Fisher matrices, we approximate the required inverse-Fisher-vector product using a streaming spectral sketch, avoiding explicit construction of the Fisher matrix. We evaluate our approach against a first-order stochastic bilevel baseline across discrete- and continuous-control environments. The results demonstrate competitive policy performance and strong reward-ranking quality, while Fisher sketching reduces curvature-storage complexity and can improve computational efficiency relative to an explicit Fisher solver.