做 LLM 微调或持续学习的团队，终于有了量化记忆容量的理论工具——MemFT 能直接帮你优化训练预算分配，建议做 LoRA 相关工作的点开看看。
该论文系统研究了 LoRA 在 LLM 微调中的参数记忆容量与动态机制，提出了 Parametric Memory Law——一个将损失减少与有效参数和序列长度联系起来的幂律关系。在 token 级别，研究发现预测概率 p > 0.5 是贪心解码下逐字回忆的充分条件，存在确定性相变。基于此，作者提出 MemFT 阈值引导优化策略，动态分配训练预算给低于阈值的 token，实验表明能提升记忆保真度和效率。这项工作首次从定量角度揭示了 LoRA 的记忆极限，对持续学习和知识更新有重要指导意义。
How LoRA Remembers? A Parametric Memory Law for LLM Finetuning
Large Language Models (LLMs) must continuously learn and update knowledge to remain effective in dynamic real-world environments. While Low-Rank Adaptation (LoRA) is widely used for such memory updates, existing studies mainly rely on qualitative downstream evaluations, leaving the quantitative capacity limits and underlying dynamics of exact parametric memory largely unexplored. To bridge this gap, we employ LoRA as a controlled memory capacity probe within the latent space to systematically quantify exact parametric memory. We introduce the Parametric Memory Law, a robust power law linking loss reduction Delta L to effective parameters and sequence length. At the token level, fine-grained analysis reveals a deterministic phase transition, demonstrating that a prediction probability of p > 0.5 constitutes a sufficient condition for verbatim recall under greedy decoding. Driven by these insights, we introduce MemFT, a threshold-guided optimization strategy that dynamically redistributes the training budget toward sub-threshold tokens. Empirical evaluations demonstrate that MemFT can enhance memory fidelity and efficiency. Code will be released at https://github.com/zjunlp/ParametricMemoryLaw.