这篇论文用KL散度做密集奖励，还加了长度控制，空间推理上比GRPO更准更短，值得搞强化学习的人看看。
LenGuard-GPC是一种密集奖励框架，针对多视图空间推理中标准GRPO奖励稀疏且无法控制推理长度的问题。它通过比较标准提示与引导提示下token级预测分布的KL散度，提供密集奖励信号。同时引入分阶段长度奖励，将推理长度控制在合理范围，避免简单鼓励短回答。在6项多视图空间推理基准上，LenGuard-GPC相比原始GRPO提升了准确率，同时降低了平均响应长度。
LenGuard-GPC: Length Guarding with Guided-Prompt Consistency for Spatial Reasoning Reinforce Learning
Multi-view spatial reasoning requires vision-language models to compare visual evidence across images, align object correspondences, and infer spatial relations over long visual contexts, a setting where chain-of-thought reasoning tends to grow verbose without becoming more accurate. Reinforcement learning with verifiable rewards is a natural fit for this task, but standard GRPO reward relies on sparse outcome-level feedback and gives no signal about where a reasoning trajectory goes wrong, nor any control over its length. We propose LenGuard-GPC, a dense reward framework that addresses both problems together. For each sampled trajectory, it compares the token-wise predictive distributions under a standard prompt and a guided prompt, and uses the resulting token-sum KL divergence as a dense reward signal. Since this KL penalty accumulates over tokens and would otherwise reward shorter responses regardless of their quality, we introduce a staged length bonus that keeps reasoning length within a controlled range without simply encouraging brevity. On six multi-view spatial reasoning benchmarks, LenGuard-GPC improves accuracy over vanilla GRPO while reducing average response length.