Papers

arXiv 论文:Transformer 可对参数化 ODE 做上下文参数估计与轨迹预测

In-Context Estimation and Prediction for Parametric ODEs

Why we picked it

一篇偏理论的论文:作者证明 Transformer 可以从几段含噪轨迹里直接估出 ODE 参数并预测轨迹,还给出了误差来源的定量拆解。做动力学建模或上下文学习理论的人可以看看这套两阶段最大似然构造的证明思路。

arXiv 论文(编号 2611.21268)研究 Transformer 对参数化常微分方程(ODE)的上下文学习:在不更新参数的前提下,从带噪声的轨迹观测中估计未知系统参数。由于该估计问题通常是非凸的,作者构造了一个两阶段近似最大似然流程:先通过粗搜索获得落入局部强凸区域的初始化,再用具有局部线性收缩性的梯度下降完成精化。论文证明,单个 Transformer 通过参数共享可以同时完成参数估计和轨迹预测,预测部分由估计器与 ODE 解映射的神经近似组合而成。

Body

作者给出了非渐近保证和有限预训练 oracle 不等式,将误差来源拆分为三部分:有限预训练数据、有限上下文长度、有限规模的 Transformer 近似误差。在线性与非线性 ODE 上的数值实验显示,预训练后的 Transformer 能在推理时通过更长的上下文获得更准的结果,无需重新训练权重。相比以往上下文学习理论多针对凸统计问题或泛函逼近保证,这项工作把分析推进到非凸的动力学系统参数估计场景。

Original · arXiv cs.AI

In-Context Estimation and Prediction for Parametric ODEs

Transformers have demonstrated the ability to perform in-context learning, that is, to adapt to new tasks from a few examples without updating their parameters. Existing theory has largely focused on emulating gradient-based algorithms for convex statistical problems or establishing general function approximation guarantees without exploiting the underlying statistical structure. We study in-context parameter estimation and trajectory prediction for parametric ordinary differential equations (ODEs), where estimation is generally nonconvex. Under suitable identifiability and regularity conditions, we construct a transformer that estimates unknown system parameters from noisy trajectory observations through a two-stage approximate maximum likelihood procedure: a coarse search provides an initialization within a locally strongly convex region, followed by gradient descent with local linear contraction. For prediction, we compose this estimator with a neural approximation of the ODE solution map. We further show that, through parameter sharing, a single transformer can perform both parameter estimation and trajectory prediction simultaneously. For these constructions, we establish nonasymptotic guarantees and finite-pretraining oracle inequalities that separate errors due to finite pretraining data, finite context length, and finite-sized transformer approximation. Numerical experiments on linear and nonlinear ODEs further show that pretrained transformers can benefit from longer inference contexts without updating their weights.

Industry Coordinates & Source Verification

This report is categorized under AITOP's "Papers" desk. Key entities and technical topics under continuous tracking include Transformer, ODE, in-context learning.

Source attribution: curated from arXiv cs.AI (@Abdullah Ahmed, Yun Yang).