这篇论文提出了PQLRM，把Pareto Q-Learning和奖励机器结合起来，在多目标任务里比基线收敛更快，还能找到普通方法找不到的最优策略。
PQLRM算法将Pareto Q-Learning与奖励机器（Reward Machines）结合，维护向量化Q估计来逼近Pareto前沿。实验表明，相比朴素PQL基线，PQLRM在奖励机器编码的非马尔可夫任务中收敛更快。它还能合成QRM无法获得的Pareto最优策略，提升了多目标强化学习的样本效率。
Pareto Q-Learning with Reward Machines
We present Pareto Q-Learning with Reward Machines (PQLRM), a multi-objective reinforcement learning algorithm for tasks whose reward structure is specified by a set of reward machines (RMs). PQLRM combines Pareto Q-Learning (PQL), which maintains sets of vector-valued Q-estimates to approximate the Pareto front, with enhancements from Q-Learning with Reward Machines (QRM), which exploits the factored automaton structure of the reward signal. This yields a multi-policy algorithm that remains sample-efficient under non-Markovian, RM-encoded rewards. Experimental trials show that PQLRM converges faster than a naive PQL baseline applied to the cross-product MDP and can synthesize Pareto-optimal policies that QRM cannot.