数值预测是数学推理和代码生成的基础能力，DEL直接改进了LLM对数字的学习效果。做数学推理或代码生成模型训练的团队，值得关注这个新损失函数，它简单有效且开源可用。
论文提出Digit Entropy Loss（DEL），一种用于大语言模型（LLM）数值学习的新损失函数。现有方法如Number Token Loss和Discretized Distance Loss分别导致数字分布过尖锐或过平坦，而DEL通过将无监督熵优化改造为有监督形式，并摒弃数值距离项，解决了这一问题。DEL支持整数、小数和小数点，将学习目标从单个数字扩展到浮点数域。在CodeLlama、Mistral、DeepSeek和Qwen-2.5等四个代表性LLM上的七个数学推理基准测试中，DEL在预测准确性和数值距离上均优于现有方法。
DEL: Digit Entropy Loss for Numerical Learning of Large Language Models
Number prediction stands as a fundamental capability of large language models (LLMs) in mathematical problem-solving and code generation. The widely adopted maximum likelihood estimation (MLE) for LLM training is not tailored to number prediction. Recently, penalty-driven approaches, e.g., Number Token Loss and Discretized Distance Loss, introduce an inductive bias of numerical distance but induce over-sharpened and over-flattened digit distributions, respectively. In this paper, we make an in-depth analysis on LLM numerical learning, and show that existing numerical learning methods conceptually follow a criterion-distance formulation, where the criterion term represents optimization pattern and the distance term instills geometric prior. Consequently, we present Digit Entropy Loss (DEL) for auto-regressive numerical learning, which reformulates the conventional unsupervised entropy optimization in three key designs: leveraging digit conditional probability and binary cross-entropy to guide the entropy optimization into a supervised manner; deprecating the distance term to bypass the issue of numerical distance; and generalizing the integer-based numerical learning to floating-point number optimization, enabling more accurate number prediction. Our DEL formulation can incorporate integers, decimals, and decimal points, expanding the learning objective from a single digit to the floating-point number domain. Experiments conducted on seven mathematical reasoning benchmarks with four representative LLMs, including CodeLlama, Mistral, DeepSeek, and Qwen-2.5, demonstrate that DEL consistently outperforms its counterparts in both overall prediction accuracy and numerical distance. Source codes are at https://github.com/PolyU-VCLab/DEL