香农缩放定律统一解释了LLM训练中的非单调退化现象，做模型缩放和训练优化的研究者可以直接用这个框架预测性能拐点，避免盲目增加计算量。
现有的大语言模型缩放定律（如单调幂律）无法解释灾难性过训练和量化退化等非单调现象。研究者提出香农缩放定律，将LLM训练建模为噪声信道上的信息传输，基于香农-哈特利定理，将模型参数映射为信道带宽，训练token映射为信号功率。该理论揭示了LLM的香农容量：若缩放模型或数据时未保持足够信噪比，噪声放大将导致性能从单调提升转为U形退化。在Pythia和OLMo2上的实验验证了该定律，其预测准确率优于经典缩放定律，并能外推到未见模型。
LLMs as Noisy Channels: A Shannon Perspective on Model Capacity and Scaling Laws
Existing scaling laws for Large Language Models (LLMs), predominantly monotonic power laws, fail to explain emerging non-monotonic phenomena such as catastrophic overtraining and quantization-induced degradation, where performance deteriorates despite increased compute. We propose the Shannon Scaling Law, a unified theoretical framework that models LLM training as information transmission over a noisy channel, grounded in the Shannon-Hartley theorem. By mapping model parameters to channel bandwidth and training tokens to signal power, our formulation explicitly captures the interaction between learning signal and intrinsic noise. This perspective reveals a fundamental Shannon capacity for LLMs: scaling model size or data without preserving a sufficient signal-to-noise ratio (SNR) inevitably amplifies noise, inducing a transition from monotonic improvement to U-shaped performance degradation. We validate our theory through experiments on Pythia and OLMo2 under perturbations, including Gaussian noise, quantization and supervised fine-tuning on math, QA and code tasks. The Shannon Scaling Law consistently outperforms classical scaling laws and recent perturbation-aware laws, achieving strong $R^2$ scores and accurately capturing loss basins missed by prior approaches. It also extrapolates: fitted on $\leq$6.9B Pythia models with $\leq$180B tokens, it predicts the unseen 12B model up to 307B tokens at pooled $R^2{=}0.847$, while monotonic baselines collapse.