Streaming联邦学习在马尔可夫数据下的泛化差距分析
On the Generalization Gap of Streaming Federated Learning with Markovian Data
理论向论文，讲清楚了联邦学习中通信越频繁泛化反而可能变差这个反直觉结论，做分布式训练的可以看看。
这篇arXiv论文研究流式联邦学习（SFL）在客户端数据服从马尔可夫过程时的期望泛化差距，使用信息论与率失真技术建立理论界。框架允许各客户端具有异构的马尔可夫动态，并在交叉熵损失下给出边界，专门针对LNTP和SANTP两种下一词预测任务。结果显示通信频率与泛化之间存在权衡：频繁聚合有助优化，但可能削弱泛化保证；时间相关性通过混合时间的平方根进入边界。作者还用TinyStories上的GPT风格语言模型预训练和时间序列预测实验验证了协作与通信频率对泛化的影响。
On the Generalization Gap of Streaming Federated Learning with Markovian Data
Streaming federated learning (SFL) allows clients to train a shared model from continuously arriving data without retaining large local datasets. Such streams often exhibit temporal dependence, yet its effect on generalization remains poorly understood. In this paper we study the expected generalization gap of the aggregated model when client data follow Markov processes, using information-theoretic and rate-distortion techniques. Our framework accommodates heterogeneous Markovian dynamics across clients and distinguishes temporal dependence from information acquired during local learning. We establish bounds under cross-entropy loss, specializing them to Linear Next-Token Prediction (LNTP) and Self-Attention Next-Token Prediction (SANTP). These results quantify the effects of client count, communication rounds, sample size, data dimension, mixing time, and context length. In particular, they reveal a communication--generalization tradeoff: although frequent aggregation may help optimization, it can weaken generalization guarantees. They also suggest a potential advantage of distributed over centralized streaming learning under a fixed total data budget. Temporal dependence enters through the square root of the mixing time. By analyzing the history-state process, we obtain refined bounds with improved context-length dependence under suitable mixing conditions. Experiments on GPT-style language-model pretraining with TinyStories and real-world time-series forecasting illustrate how collaboration and communication frequency influence generalization under temporal dependence