CHARM+：跨硬件注意力机制，比 Ring-Attention 最快提速 2.9 倍
CHARM+: Cross-Hardware Attention with Re-Merge Multistream Mechanism
训练长序列的朋友可以看看这篇：CHARM+ 把跨节点通信和计算重叠起来，双节点 NVLink 上比 Ring-Attention 快近 3 倍。
CHARM+ 是一种面向长序列分布式注意力的新方法，通过 Re-Merge 多流机制把串行通信重组为多级并行并与计算重叠。它还将跨硬件通信与硬件内部通信解耦，消除因带宽不均衡导致的同步开销。在双 NUMA PCIe 架构上，前向和反向相比 Ring-Attention 平均加速 1.3 倍和 1.7 倍；在双节点 NVLink 架构上达到 2.8 倍和 2.9 倍。论文已发布于 arXiv，编号 2611.05906。
CHARM+: Cross-Hardware Attention with Re-Merge Multistream Mechanism
As sequence lengths continue to grow, distributed attention based on sequence parallel (e.g., Ring-Attention) has become a mainstream solution for large language models (LLMs). Nevertheless, inherent communication dependencies limit computation-communication overlap, reducing GPU utilization. Moreover, in cross-hardware (NUMA or node) architectures, due to imbalanced communication bandwidth, frequent communication synchronization will limit overall performance. Based on these insights, we propose CHARM+, Cross-Hardware Attention with Re-Merge Multistream Mechanism. We reorganize sequential communications into several levels and achieve intra-level parallelism and computation overlap. Furthermore, we decouple cross-hardware communication from intra-hardware communication, eliminating synchronization overhead caused by bandwidth imbalance. Forward and backward algorithms were evaluated on multiple hardware architectures and consistently outperformed state-of-the-art methods. Compared to Ring-Attention, our forward and backward algorithms achieve average speedups of 1.3x and 1.7x on a dual-NUMA PCIe architecture, and 2.8x and 2.9x on a dual-node NVLink architecture, respectively.