用分布强化学习做多目标在轨检查任务的风险敏感规划
Risk-Sensitive Mission Planning for Multi-Target On-Orbit Inspection via Distributional Reinforcement Learning
一篇把 QR-DQN 用到太空在轨检查任务上的论文，风险规避策略能把回报方差砍一半，还用枚举法验证了接近最优前沿。
这篇 arXiv 论文提出基于 Distributional Reinforcement Learning 的在轨服务任务规划方法，使用 Quantile-Regression Deep Q-Network（QR-DQN）智能体，通过均值-方差准则在期望与方差之间权衡，决定低推进力 servicer 在近地轨道中下一个检查的目标。任务环境采用 Edelbaum 组合机动低推进转移估计建模圆形轨道，目标的风险状态会在任务过程中随机变化。在五颗目标卫星的数值实验中，风险规避智能体将回合回报方差降低约一半，并降低任务失败概率，代价是平均回报下降。通过枚举所有可行访问顺序计算的 efficient frontier 验证，学习到的策略在相同方差下平均回报接近最优前沿。
Risk-Sensitive Mission Planning for Multi-Target On-Orbit Inspection via Distributional Reinforcement Learning
Multi-target on-orbit servicing missions require planning tools that can adapt to evolving operational conditions while accounting for the outcome variability inherent to safety-critical spaceflight. This paper presents an adaptive planning approach for multi-target servicing campaigns in low Earth orbit and applies it to inspection missions flown by a low-thrust servicer. The approach is based on Distributional Reinforcement Learning, which learns the full distribution of mission returns and thereby supports explicit risk-sensitive decision-making at each planning step. Specifically, a Quantile-Regression Deep Q-Network (QR-DQN) agent selects which target to inspect next by jointly considering the expectation and the variance of the predicted return through a mean-variance criterion. The mission environment models circular orbits with a spread in inclination and uses Edelbaum's combined-manoeuvre low-thrust transfer estimates. The observation comprises the orbital elements, the visit history, the observed risk state of each target and the remaining mission-time budget. The risk state of each target changes stochastically during the campaign, making the outcome of an inspection uncertain and creating operationally meaningful distinctions between risk-neutral and risk-averse planning strategies. Because the mission-time budget generally admits only a subset of the candidate targets, the planner must also select which targets to inspect. Numerical experiments on scenarios with five target satellites show that one risk-aversion weight moves the policy across a range of mean return and return variance, over an unchanged environment and reward. Risk-averse agents reduce the variance of the episodic return by approximately one half relative to risk-neutral agents and lower the probability of a mission loss, at a cost in mean return. To validate the learned policies, the efficient frontier, which gives the highest mean return attainable at each level of return variance, is computed exactly by enumerating the admissible visit orders. At equal variance, the learned policies lie close to this frontier in mean return, and the risk-averse policies have a lower probability of a mission loss than the frontier policies.