Conduit：面向分布式强化学习的经验数据平面运行时
Conduit: An Experience Data Plane for Distributed Reinforcement Learning
分布式 RL 训练卡在经验搬运上？这篇论文的 Conduit 把经验路径延迟最多降 97%，还能扩到 1,024 GPU。
arXiv 论文提出 Conduit，一个框架无关的运行时，把分布式强化学习的经验管理变成显式的系统优化问题。其核心抽象 Experience Data Plane（EDP）将经验的摄取、放置与交付拆成独立控制点，与框架执行逻辑解耦。Conduit 引入容量受限、带宽感知的经验放置，在异构互连和显存约束下把经验状态分布到 CPU/GPU 内存层级与节点间，并配合延迟感知调度降低经验路径暴露延迟。集成到 RLlib 后无需改动框架执行逻辑，经验路径暴露延迟最多降低 97%，端到端迭代延迟最多降低 38%，可扩展至 1,024 GPU 且保持收敛。
Conduit: An Experience Data Plane for Distributed Reinforcement Learning
Distributed reinforcement learning (RL) scales training by parallelizing actors and learners around an Experience Buffer. As RL workloads grow, however, the buffer becomes more than a replay queue: it is the storage substrate of a large-capacity, latency-critical experience path that every iteration traverses to move, transform, sample, and batch experiences before learner updates can begin. Existing RL systems embed this path inside framework control flow or expose it as a request-driven buffer service, leaving experience placement fixed and experience-path work difficult to schedule independently as a runtime-level optimization target. We present Conduit, a framework-agnostic runtime that exposes RL experience management as an explicit systems optimization problem. At its core is the Experience Data Plane (EDP), a runtime abstraction that separates RL experience-handling semantics from framework-specific execution logic by exposing experience ingestion, experience placement, and experience delivery as explicit control points. Built on EDP, Conduit introduces capacity-constrained, bandwidth-aware placement, which distributes experience state across CPU/GPU memory tiers and nodes under heterogeneous interconnect and device-memory constraints, and latency-aware scheduling, which controls when experience-path handling runs to reduce exposed experience-path latency while preserving RL semantics. Integrated with RLlib without changing its framework execution logic, Conduit reduces exposed experience-path latency by up to 97% and end-to-end iteration latency by up to 38%, scales to 1,024 GPUs, and preserves convergence.
