这篇论文提出了首个可实现的泛函梯度下降算法，能自适应调整梯度表示，理论上有收敛保证，实验上比传统FGD和神经网络更快更准。
这篇论文提出了一个新型的泛函梯度下降（FGD）算法，该算法在优化过程中自适应调整泛函梯度的表示，解决了固定近似引入误差的问题。作者证明，在光滑损失条件下该算法收敛到平稳点，在附加Polyak-Lojasiewicz条件时收敛到全局最小值，这是首个在一般环境下具有此类保证的可实现FGD方法。在回归、偏微分方程数值求解和现代计算机视觉任务中，该方法在效率和准确度上均优于固定近似FGD和神经网络基线。
Functional Gradient Descent with Adaptive Representations
Functional optimization problems are typically solved by optimizing the parameters of a fixed representation, such as a neural network, resulting in highly nonconvex losses that complicate both training and theoretical analysis. An interesting alternative is functional gradient descent (FGD), that is, gradient descent directly in function space, which benefits from strong convergence results and admits a clean theory. However, FGD is difficult to implement in practice because functional gradients are infinite-dimensional, and thus cannot be fully computed nor stored in memory. Existing implementations therefore rely on fixed approximations, which introduce approximation error. We propose a new, theoretically-grounded FGD algorithm that adapts the representation of the functional gradients over the course of optimization. By explicitly incorporating this approximation into the analysis, we establish convergence to a stationary point (for smooth losses) and to a global minimizer (under smoothness + a Polyak-Lojasiewicz-type condition) regardless of our approximations. To the best of our knowledge, this is the first implementable FGD method with such guarantees in a general setting. We demonstrate the effectiveness of our method on regression, numerical solution of PDEs, and modern computer vision. Across settings, our method consistently outperforms both FGD with fixed approximations and neural network baselines in efficiency and accuracy.