GIST：面向离线 RLVR 的分层训练数据选择方法
GIST: Gibbs-Targeted Hierarchical Data Selection for Offline RLVR
一篇教你离线 RLVR 该挑哪些回答、哪些 token 来训的论文，在 Qwen3-0.6B 的 GSM8K 上比七个基线都强，还省了三分之二的监督 token。
arXiv 论文提出 GIST（Gibbs-Targeted Hierarchical Data Selection），一种针对离线 RLVR 训练的两层数据选择方法。作者用 KL 正则化 RLVR 的 Gibbs 最优策略分析策略与目标之间的双向 KL 散度，发现正确回答主要通过 forward 项集中在低概率 token 上，错误回答通过 reverse 项集中在策略高频生成的高概率 token 上。GIST 利用采样温度选择正确回答和接近策略众数的错误回答，并只对失配所在的 token 施加监督。实验在 GSM8K、MATH-500 和 MBPP 上提升了三个模型在两种离线目标下的平均准确率，在 Qwen3-0.6B/GSM8K 对照实验中超过七个选择基线，且比监督全部候选 token 少用 66.6% 的 token。
GIST: Gibbs-Targeted Hierarchical Data Selection for Offline RLVR
Offline reinforcement learning with verifiable rewards (RLVR) trains on a fixed pool of sampled responses and typically supervises every token of every response, although responses and tokens are not equally informative. We study which ones to supervise through the Gibbs-optimal policy of KL-regularized RLVR, which offline training targets, and measure the mismatch between a policy and this target with a bidirectional KL divergence. At the reference policy, this mismatch factorizes exactly and separates the two outcomes: correct responses contribute mainly through the forward term, concentrated on their low-probability tokens, and incorrect responses through the reverse term, concentrated on the responses the policy generates often and on their high-probability tokens. Based on this analysis, we propose GIST (Gibbs-Targeted Hierarchical Data Selection), a two-level selection method that approximates these principles with signals available before training. GIST uses the sampling temperature to select correct responses on-policy and incorrect responses near the policy's mode, and it uses the initial policy's token probabilities to supervise only the tokens where the mismatch lies. Applied to two offline training objectives, GIST raises the average accuracy over GSM8K, MATH-500, and MBPP for all three models under both objectives. In a controlled comparison on Qwen3-0.6B/GSM8K, it outperforms seven selection baselines that choose from the same candidates while supervising 66.6% fewer tokens than training on all candidates of the same prompts.
