这篇论文的KGRL算法把领域知识直接融入强化学习训练，比传统基线更省样本、回报更高，适合做PAMDP相关研究的人参考。
本文研究参数化动作马尔可夫决策过程（PAMDP）中的强化学习问题。现有算法通常采用单次估计器确定参数，导致训练样本效率低下。作者提出KGRL算法，利用Datalog知识库中的领域知识修剪非适用动作并约束参数空间，再通过梯度参数精化环优化参数。在多个PAMDP基准环境中，KGRL在样本效率和情节回报上均优于现有基线。
Knowledge- and Gradient-Guided Reinforcement Learning for Parametrized Action Markov Decision Processes
In this paper, we study Reinforcement Learning in Parametrized Action Markov Decision Processes (PAMDP), where each decision consists of a symbolic action and numerical parameters. In such settings Reinforcement Learning algorithms typically determine parameters with one-shot estimators, which makes their training sample inefficient. Though in most PAMDP environments explicit but incomplete knowledge (e.g., rules, safety constraints, or expert heuristics) is available, it is rarely directly used to increase the sample-efficiency of training Reinforcement Learning agents. We step into this gap and propose our novel Neuro-Symbolic Knowledge- and Gradient-Guided Reinforcement Learning (KGRL) algorithm. KGRL uses domain knowledge in a Datalog knowledge base to derive the set of applicable actions and feasible parameters for a given state. This allows it to prune non-applicable actions from the decision-space and constrain the parameter spaces of the remaining actions. We then use a gradient-based parameter refinement loop to estimate the optimal parameters during training and deployment of the agent. By recording activated rules along the trajectory, KGRL additionally provides local procedural explanations on the pruning of actions and constraining of parameters. Overall, KGRL guides the agent's exploration and deployment toward feasible and constraint-aware decisions, while increasing sample efficiency during training. KGRL outperforms state-of-the-art RL baselines for PAMDPs in both, sample efficiency and episodic return.