这篇论文提出了HyGAE方法，让VLM智能体在多个回合的决策任务里表现更好，平均成功率91%，比别的方法高了10%，值得搞强化学习和多模态的研究者看看。
HyGAE是一种针对视觉-语言模型（VLM）智能体的强化学习框架，提出混合优势估计（Hybrid Advantage）和统一评价器（Unified Critic），同时优化令牌级和回合级目标。在五个多回合决策环境中的评估显示，HyGAE的平均成功率达到91%，相比其他方法提升了10%。理论分析表明，混合优势和回报的精确解析形式对优化至关重要。
Hybrid Advantage Estimation with Unified Critic for VLM Agentic Reinforcement Learning
Large Vision-Language Models (VLMs) now act as agents in interactive environments, where success requires coherent reasoning and decision-making across turns. Although end-to-end training in agentic environments can improve such multi-turn decision-making abilities, current methods mainly rely on either token-wise optimization over concatenated token trajectories or turn-wise optimization with uniform within-turn credit. In this work, we establish theoretical formulations for the two levels of optimization and derive a hybrid advantage that serves both objectives. Furthermore, with an appropriate choice of discount factor and learning target, we prove that a unified critic model can estimate values for both turn-wise and token-wise. As such, we propose HyGAE, an actor-critic framework that jointly optimizes token- and turn-level objectives with the hybrid advantage and unified critic. We conduct extensive evaluations of HyGAE across five multi-turn decision-making environments, where it achieves an average success rate of 91% and a significant improvement of 10% over other methods. Furthermore, we provide an in-depth analysis showing that the exact analytic form of the hybrid advantage and return is crucial for optimization. Project Page: https://wx-zhang.github.io/hygae-web/.