Atomic MathReasoner:用原子推理技能组合提升 LLM 数学能力
Atomic MathReasoner: Instance-Level Composition of Atomic Reasoning Skills for Mathematical LLMs
把数学推理拆成 LoRA 专家库再按题路由,思路和传统 MoE 不太一样,用的是无梯度优化加 NLL 打分。7B 模型逻辑推理提了 7 个点,做模型合并或路由的人可以看看方法细节。
一篇 arXiv 论文提出 Atomic MathReasoner(AMR),一个实例级框架,把数学推理拆解为原子技能再按题目组合,而非作为整体能力训练。AMR 先训练一组能力专属的 LoRA 专家,然后在留出的验证子集上做无梯度优化,发现多个低损失组合原型;再用参考解答的 token 级负对数似然评估每个原型的偏好,通过置信度边际过滤模糊分配,最后把伪标签蒸馏进一个轻量路由器。推理时路由器为每道题选择合适的组合原型,避免昂贵的在线优化。
在 AtomicMath 基准上,Qwen2.5-Math-Instruct-7B 的 Level 1 平均分项准确率从 64.4% 提到 68.1%,Level 2 从 59.3% 提到 63.7%,平均逻辑推理从 39.4% 提到 46.8%,通用型与数学专用型 LLM 上均有一致提升。对做模型组合与 MoE 路由的开发者来说,这套“专家库 + 无梯度原型发现 + 蒸馏路由”的流程值得参考。
Atomic MathReasoner: Instance-Level Composition of Atomic Reasoning Skills for Mathematical LLMs
Most existing approaches enhance mathematical reasoning in large language models (LLMs) as a monolithic capability, despite the fact that different mathematical problems require different reasoning skills. Recent work has explored a more fine-grained perspective by decomposing mathematical reasoning into atomic skills and developing specialized modules for individual capabilities. However, how to effectively utilize these atomic abilities remains underexplored. To address this challenge, we propose Atomic MathReasoner (AMR), an instance-level framework for composing atomic reasoning skills. AMR first trains a library of capability-specific LoRA experts and performs gradient-free optimization over held-out validation subsets to discover a diverse set of low-loss composition prototypes. It evaluates the preference of each prototype using the token-level negative log-likelihood of reference solutions, filters ambiguous assignments through a confidence margin, and distills the resulting pseudo-labels into a lightweight router. During inference, the router selects an appropriate composition prototype for each problem, enabling problem-specific utilization of atomic skills without costly online optimization. Experiments on the AtomicMath benchmark across general and mathematics-specialized LLMs demonstrate consistent improvements. On Qwen2.5-Math-Instruct-7B, AMR increases average field accuracy from 64.4% to 68.1% at Level 1 and from 59.3% to 63.7% at Level 2, while improving average logical reasoning performance from 39.4% to 46.8%. These results demonstrate that effectively composing specialized atomic skills provides a promising approach to strengthening mathematical reasoning in LLMs.
This report is categorized under AITOP's "Papers" desk, focusing on China's AI industry and domestic foundation model ecosystem. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include Atomic MathReasoner, Qwen2.5-Math-Instruct-7B, LoRA, math reasoning.
Source attribution: curated from arXiv cs.LG (@Jiashu Li, Tianyu Gao, Qinglong Liu, Jinping Wang, Ye Wei, Zhiqiang Gao).