BAQ-TD 算法证明限速信道下 TD 学习可保持收敛速率
Adaptive Quantization Preserves Fast Rates for TD Learning over Rate-Limited Channels
做分布式或边缘端强化学习的人可以看看这篇。它证明了每步只需传输 O(d log(T)) 比特就能保住未量化 TD 的收敛速率，比之前的结果有指数级改进。 batching 加自适应量化的组合思路也值得借鉴到自己的通信受限训练管线里。
arXiv 论文提出 Batched Adaptively Quantized TD（BAQ-TD），研究智能体通过有限容量信道向服务器传输量化后的时序差分（TD）更新方向的策略评估问题。算法用 batching 降低传输信息的方差，再用自适应量化编码增量信息，并给出了高概率有限时间误差界。
核心结论：对于 d 维值函数参数，在长度为 T 的时间范围内总共传输 O(d log(T)) 比特，即可在有限信道容量以上保持未量化 TD 的收敛速率（相差对数因子），达到 Õ(1/T) 的阶最优速率，相比此前工作是通信复杂度上的指数级改进。
该结果面向带通信约束的强化学习场景，如边缘设备与中心服务器带宽受限的分布式训练，说明统计效率不必以无限带宽为前提，为量化强化学习的理论分析提供了新基准。
Adaptive Quantization Preserves Fast Rates for TD Learning over Rate-Limited Channels
Motivated by reinforcement learning subject to communication constraints, we study a policy evaluation problem with linear function approximation over a rate-limited channel. In our setup, an agent interacting with an unknown environment (modeled as a Markov Decision Process) transmits quantized estimates of a temporal-difference (TD) learning update direction to a server over a finite-capacity channel. Our goal is to investigate to what extent one can preserve the statistical convergence rates achievable when the channel has infinite capacity. To that end, we propose a new algorithm called Batched Adaptively Quantized TD (BAQ-TD), which uses batching to reduce the variance of the transmitted information, and adaptive quantization to encode innovations. We prove a high-probability finite-time error bound for BAQ-TD, and show that, above a certain finite channel capacity, BAQ-TD preserves the convergence rate of unquantized TD up to logarithmic factors. Specifically, for a $d$-dimensional value function parameter, we establish that transmitting a total of $\mathcal{O}(d \log(T))$ bits over a horizon of length $T$ suffices to achieve the order-optimal rate of $\tilde{\mathcal{O}}(1/T)$, marking an exponential improvement in communication-complexity over prior work.
This report is categorized under AITOP's "Papers" desk. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include TD学习, 强化学习, 量化, BAQ-TD.
Source attribution: curated from arXiv cs.LG (@Sreejeet Maity, Aritra Mitra).