做 RL 训练或大模型推理的团队终于有了解决策略偏移的实用方案——TITO 让每个 token 都对齐，计算量还能省 10 倍，搞 Agent 训练的开发者值得点开看看。
LMSYS 在博客中介绍了 Token-In-Token-Out (TITO) 技术，用于解决强化学习中推理与训练 token 不一致导致的策略偏移问题。TITO 通过确保训练器使用推理引擎产生的精确 token，使每个 token 保持在策略上，从而提升训练效率。该技术将每个任务视为一个样本而非每个回合，在 30-50 回合的轨迹上可节省约 10 倍计算量。Miles 框架通过推理会话服务器、追加式 token 缓冲区、可插拔 TITO tokenizer 和 TokenSeqComparator 等组件实现 TITO。该技术已支持 Qwen3、GLM、Kimi-K2、Nemotron、Minimax 和 DeepSeek 等模型系列。
📝 New blog: No Token Left Behind: Demystifying Tok…
📝 New blog: No Token Left Behind: Demystifying Token-In-Token-Out in Miles
In agentic RL, a rollout is a chain of model calls, tool outputs & resumed turns. Token-In-Token-Out (TITO) ensures the trainer evaluates the exact tokens the inference engine produced — break it, and training silently drifts off-policy.
Why it matters: 📦 One sample per task, not per turn: ~10× less compute on 30–50 turn trajectories 🎯 Keeps every token on-policy
How Miles enforces it: 1️⃣ Inference session server: one append-only token buffer per trajectory 2️⃣ Append-only at 3 levels: messages, template rendering, tokens 3️⃣ Pluggable TITO tokenizer: incremental tokenize + per-model splice patches 4️⃣ TokenSeqComparator: verifies every rollout stays bit-perfect
Supports Qwen3, GLM, Kimi-K2, Nemotron, Minimax & DeepSeek families.