BARTER：用标准稠密算子实现紧凑动态稀疏训练，L40 提速 6.29 倍
BARTER: Compact Dynamic Sparse Training with Standard Dense Kernels
BARTER 不用专用稀疏算子，靠张量紧凑化把训练提速 6.29 倍，显存省 81%，iPhone 上也能跑。
arXiv 论文提出 BARTER，一种结构化动态稀疏训练（DST）框架，不依赖专用稀疏算子，仅用标准稠密算子就能降低训练成本。它在普通反向传播中只收集活跃单元的统计量，据此对注意力头和前馈通道做剪枝与克隆，并在固定全局预算下动态重分配各层容量。每次拓扑更新时，BARTER 会缩放受影响的参数张量并重映射优化器状态，后续训练直接在紧凑模型上进行。在两个图像分类基准、0.1 到 0.7 密度范围内对 Transformer 视觉模型的训练与微调实验显示，Top-1 精度与全张量 DST 基线相当。NVIDIA L40 上实测训练段最高加速 6.29 倍、峰值显存占用最多降低 81.3%，iPhone 13 Pro Max 训练基准也取得 2.78 倍加速。
BARTER: Compact Dynamic Sparse Training with Standard Dense Kernels
Dynamic sparse training (DST) aims to reduce training costs while maintaining accuracy. Existing DST methods mask out parameters but keep matrix dimensions unchanged. Training speedups therefore depend on specialized support, such as sparse kernels. We introduce BARTER, a structured DST framework that reduces training cost using standard dense kernels by keeping training tensors physically compact while dynamically reallocating capacity across layers. Unlike existing works that require computing gradients for all parameters, BARTER instead uses statistics collected only from active units during ordinary backpropagation to prune and clone attention heads and feed-forward channels under separate fixed global budgets. At each topology update, it resizes the affected parameter tensors and remaps their optimizer states, so subsequent training steps operate directly on the compact model. Training and fine-tuning experiments with Transformer-based vision models on two image classification benchmarks at densities from $0.1$ to $0.7$ show competitive Top-1 accuracy relative to full-tensor DST baselines. On an NVIDIA L40, BARTER achieves speedups of up to $6.29\times$ in measured training segments and reduces peak allocated GPU memory by up to $81.3\%$ relative to these implementations. An iPhone 13 Pro Max training benchmark additionally shows a $2.78\times$ speedup over the full-tensor DST baseline.
