OpenAI研究团队用计算器工具+强化学习，把数学推理准确率提升30个百分点，方法开源可复现。
研究分析了大语言模型在Countdown任务中的计算错误问题。构建了监督微调数据集教授模型工具使用模式。应用RLOO、RLOO++、GRPO和DAPO等强化学习方法。在1024题的独立测试基准上，Tool-DAPO将pass@1从35.8%提升至66.0%。强化学习鼓励更有效的工具使用，即使只提供最终答案奖励。
Learning to Use Tools: Reinforcement Learning for Tool-Integrated Mathematical Reasoning
Current large language models (LLMs) increasingly benefit from external tool integration, especially for tasks requiring reliable computation and verification. Motivated by this, we study calculator tool calling for improving mathematical reasoning on the Countdown task. We first analyze reasoning failures and find that calculation errors account for a substantial portion of incorrect responses. We then construct supervised fine-tuning datasets to teach the model useful tool-use patterns and how to interpret returned outputs. Building on this tool-formatted policy, we apply several on-policy reinforcement learning methods, including RLOO, RLOO++, GRPO, and DAPO, using automatically verifiable final-answer rewards. To enable a more reliable evaluation, we construct a fresh 1,024-problem held-out Countdown benchmark with no exact overlap with the training data. Our results show that calculator tool integration consistently improves both SFT and RL baselines, yielding roughly 10 percentage-point gains across pass@k. Among the RL methods, Tool-DAPO achieves the strongest performance, improving pass@1 from 35.8% for Tool-SFT to 66.0%. Further analysis shows that RL encourages more effective tool use even when only final-answer rewards are provided. These findings suggest that tool integration reduces arithmetic and verification errors, while RL increases the probability of correct reasoning traces.