做小模型算术推理的团队可以借鉴这种教学法思路——用人类教学步骤替代复杂强化学习,86M参数就能达到大模型水平,值得一试。
该研究借鉴印尼GASING算术教学法,将左到右的计算步骤序列化为自然语言思维链(CoT)监督数据,训练了一个86M参数的GPT-2模型。模型仅使用下一个词预测目标,无需强化学习或奖励优化,在保留问题上达到80%以上准确率,性能可与更大模型竞争。通过注意力掩码、残差流探测等机制分析发现,模型先内化程序化路径,随后发展出类似心算的关联检索能力。这项工作表明,基于教学法的针对性训练能以较小规模实现强算术能力。
Arithmetic Pedagogy for Language Models
We investigate whether methods of human mathematics pedagogy can guide the training of language models toward arithmetic reasoning. Building on the GASING method -- an Indonesian pedagogy that solves basic arithmetic through a left-to-right procedure aligned with the causal order of token generation -- we operationalize each operation as a computational procedure whose execution trace is serialized into natural-language Chain-of-Thought (CoT) supervision. A small GPT-2 decoder (86M parameters) with a syllabic-agglutinative TOBA tokenizer for Indonesian is trained from scratch on this data using only a next-token prediction objective, without reinforcement learning or reward-based optimization. Monitoring training reveals three distinct learning phases, and mechanistic analyses -- attention-masking interventions on the CoT information graph, residual-stream probing, and logit-lens inspection -- show that the model first internalizes a procedural pathway and subsequently develops an associative, ``mental-arithmetic'' capacity that retrieves intermediate results without explicit step-by-step computation. The trained model reaches over 80% accuracy on held-out problems and attains competitive performance against substantially larger language models, indicating that targeted, pedagogically grounded training can yield strong and economical arithmetic capability at small scale.