数值线性代数如何支撑 LLM：一篇面向数值方法专家的综述
The Numerical Linear Algebra of Large Language Models
写给数值计算背景的人看的 LLM 综述，从 QR 算法一路讲到 Transformer 里的矩阵方法，能把两个领域的桥接讲清楚。
arXiv 上发布了一篇综述文章，回顾数值线性代数（NLA）从 1950 年代求解特征值问题的 LR 和 QR 算法，到稀疏线性系统求解的演进历史。文章指出当今 LLM 的底层依赖矩阵与张量方法，NLA 正再次处于技术变革的中心。综述有两个目标：一是用数值方法专家能理解的方式讲清 LLM 核心概念，二是梳理 LLM 技术中关键的 NLA 概念及其近期贡献。文章编号为 arXiv:2610.04631v1。
The Numerical Linear Algebra of Large Language Models
Numerical Linear Algebra (NLA) has consistently played a vital role in advancing science by providing tools to solve fundamental problems encountered in scientific and engineering applications. Over the decades, it has continually evolved to meet the demands driven by successive waves of scientific discovery. For instance, during the 1950s and 1960s, substantial efforts were devoted to developing methods for solving eigenvalue problems that emerged from the rapidly growing field of aerodynamics. This led to the discovery of the LR and QR algorithms. Later the attention turned to the solution of sparse linear systems that were common in applications like computational aerodynamics. Today we are experiencing yet another wave of major scientific advancement and NLA is once more at the heart of its development. This Machine Learning (ML) wave is proving to be utterly disruptive in science and engineering. Many tools in ML particularly Large Language Models (LLMs) are grounded in matrix and tensor methods. As we are approaching Artificial General Intelligence (AGI), it is clear that matrix methods will be called to play an even more significant role. For the numerical linear practitioner the speed of the current change makes it particularly challenging to adapt. This is a survey article that centers on machine learning techniques, with a particular focus on large language models. It has two main objectives. The first is to clarify the core concepts behind Large Language Models in a manner accessible to specialists in numerical methods. The second is to examine the key Numerical Linear Algebra concepts employed by LLM techniques, while also highlighting several significant recent contributions of NLA to the field.
