分层策略:用多个 LoRA 适配器优化推理时 pass@k 采样
Learning Stratified Policies for Inference-Time Objectives
做推理时采样或 best-of-n 的朋友可以看看这篇:不重复采一个模型,而是挂多个 LoRA 各出一个答案,理论证明了误差能降为零,pass@k 在多个推理任务上稳定跑赢单策略。想挤榨采样预算的话值得照着实现试试。
arXiv 论文(编号 2611.23154)提出 stratified policies(分层策略),将推理时的 k 次采样预算分配给 k 个不同的学习策略,而不是反复从单一策略采样。实现方式是在共享的语言模型主干上挂载 k 个独立的 LoRA 适配器,每个策略贡献一个回答,并以 pass@k 为目标联合优化。理论层面,作者在线性设置下证明:在单个 LoRA 都存在严格正预测误差的场景中,一组 LoRA 可以把 best-of-k 近似误差降到零。
实验显示,在多个推理任务上,分层策略的 pass@k 一致优于单策略基线。核心结论是:把采样预算分散到互补的策略上,比从单一策略做 i.i.d. 重复采样更能提升推理时扩展的效率。
Learning Stratified Policies for Inference-Time Objectives
Inference-time scaling improves the performance of large language models by sampling multiple responses and evaluating their collective utility. However, many existing approaches obtain these samples by repeatedly sampling from a single policy. In this work, we study if the sample budget at inference time can instead be allocated across a population of policies. We introduce stratified policies, where each of $k$ learned policies contributes one response to a $k$-sample inference-time objective. We realize this policy population using separate LoRA adapters on a shared language model backbone and optimize it for pass@$k$. We theoretically motivate the approach by analyzing the linear setting; a population of LoRAs can achieve zero best-of-$k$ approximation error in settings where every single LoRA has strictly positive prediction error. Empirically, we demonstrate that stratified policies consistently improve pass@$k$ over a single-policy baseline across multiple reasoning tasks. Overall, our results highlight how distributing samples across complementary policies can be a more effective use of sample budget than repeated i.i.d. sampling from a single policy at inference time.
This report is categorized under AITOP's "Papers" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include LoRA, pass@k, reasoning models, arXiv.
Source attribution: curated from arXiv cs.AI (@Mridul Mahajan, Xuezhou Zhang, Aldo Pacchiano).