Reciprocal Q-Augmentation：不共享奖励也能在博弈中学合作
Cooperation Without Shared Rewards via Reciprocal Q-Augmentation
一群各自只看自己奖励的智能体，怎么学会不互相坑？这篇论文给了个 Q 函数增强的解法，在四个经典博弈环境里跑赢了共享奖励的做法。
论文提出 Reciprocal Q-Augmentation（RQA），用隐变量对手模型增强 Q 函数，估计自身动作对其他智能体的影响。该方法不依赖其他智能体的奖励或内部状态，仅凭自身 Q 函数就能学出合作与背叛两种策略。在 Iterated Prisoner's Dilemma、Iterated Traveler's Dilemma、Iterated Stag Hunt 和 CleanUp-Lite 四个社会困境环境中，RQA 比共享奖励的全合作算法收敛更快、合作水平更高。单个 RQA 智能体在合作与背叛策略间切换，还能诱导用标准 RL 训练的对手产生合作。
Cooperation Without Shared Rewards via Reciprocal Q-Augmentation
General-sum games are more challenging than fully cooperative or adversarial settings because agents may deceive or exploit one another. Achieving high social welfare requires sustained cooperation, while a single defection can drive agents toward a welfare suboptimal Nash equilibrium. Moreover, assuming access to other agents' rewards or internal states is often unrealistic and implicitly introduces additional cooperation into the setting. To address this issue, we introduce \textit{Reciprocal Q-Augmentation} (RQA), a method that uses a latent-variable opponent model to augment the Q-function with estimates of how one's actions affects other agents. RQA allows agents to learn both cooperative and defective policies using only their own Q-function. We show that, in social dilemmas, agents that all follow the cooperative policy of RQA learn to cooperate faster and reach higher levels of cooperation than fully cooperative algorithms. Moreover, a single RQA agent that switches appropriately between its cooperative and defecting policies can induce cooperation in agents trained with standard RL algorithms, which would otherwise end up defecting. We evaluate our approach on the Iterated Prisoner's Dilemma, Iterated Traveler's Dilemma, Iterated Stag Hunt, and \texttt{CleanUp-Lite}. Our experiments show that the algorithms implementing RQA outperforms fully cooperative methods with shared rewards and enables cooperation with algorithms that otherwise fail to reach high social welfare.
