这篇论文教你用强化学习把难题拆成小模块再拼起来，Qwen和Code World Model上测试比传统RL省50倍算力，还能解原本解不出的题。
DecompRL是一种强化学习算法，专门训练大语言模型（如Qwen 2.5 7B、Code World Model 32B）将复杂问题分解为可独立求解的子函数并重新组合。通过重组n个模块的k种实现，可产生最多k^n个候选解，将GPU推理瓶颈转移到廉价CPU评估，GPU token成本降低约50倍。在LiveCodeBench和CodeContests基准上，当每个问题的推理token超过10^5时，DecompRL显著优于标准RL和多样性优化RL基线，能解决标准生成方法无法触及的问题。
DecompRL: Solving Harder Problems by Learning Modular Code Generation
How can Large Language Models (LLMs) solve problems they currently cannot? Repeated sampling scales test-time compute but GPU cost grows linearly with attempts, while reinforcement learning (RL) with verifiable rewards improves single-attempt accuracy at the expense of sample diversity. Both strategies ultimately fail when the base policy has near-zero probability of producing a correct solution: no amount of sampling or gradient signal can overcome a search space that is simply too large. We take a different approach: rather than sampling harder, we make the task easier by decomposing problems into smaller, independently solvable sub-functions whose implementations can be recombined. Since off-the-shelf models are not trained for this modular generation, we introduce DecompRL, an RL algorithm that explicitly learns to decompose and implement hierarchical code structures. Recombining $k$ implementations of $n$ modules yields up to $k^{n}$ candidate solutions, shifting the bottleneck from GPU inference to cheap CPU evaluation and cutting GPU token cost by $\sim$50$\times$. On LiveCodeBench and CodeContests (Qwen~2.5~7B, Code World Model~32B), DecompRL outperforms standard and diversity-optimized RL baselines beyond $10^5$ tokens per problem, solving problems that standard generation cannot reach.