Accelerating Sharded Data Parallelism with Federated Learning
Accelerating Sharded Data Parallelism at Scale with Federated Learning
This research provides a concrete method for improving the efficiency of training large AI models, which is relevant for anyone working with or studying large language models.
This research introduces FL+FSDP and FL+HSDP hybrid algorithms that combine sharded data parallelism with federated learning principles to reduce communication overhead in large-scale AI model training.
A Llama3.1 8B model pre-trained on 512 A100 GPUs showed FL+FSDP and FL+HSDP achieved up to 8.
04x faster data processing and 4.48 lower evaluation perplexity compared to standard methods.
Accelerating Sharded Data Parallelism at Scale with Federated Learning
The symbiotic scaling of artificial intelligence models and high-performance computing systems continually creates algorithmic challenges in their convergence. Foundation models (FMs) are a crucial example, requiring months-long training on thousands of cutting-edge GPUs. Sharded data parallelism (DP) is the dominant strategy to accelerate such computations by splitting data and models across multiple GPUs. However, it incurs prohibitive communication overhead when deployed at scale, particularly on multi-tier interconnects with heterogeneous performance. Inspired by the efficient communication principles of federated learning (FL), this work introduces two hybrid algorithms - FL+FSDP and FL+HSDP - interleaving sharded DP with FedAvg-style aggregations. Such approaches decouple large DP deployments into smaller, loosely-coupled federation groups, requiring minimal inter-group traffic while keeping the global batch size bounded by the groups' size. Formal analysis of communication costs and experimental validation prove their scalability and flexibility. A Llama3.1 8B pre-training on 512 A100 GPUs shows that, under identical hyperparameters, FL+FSDP and FL+HSDP achieve up to 8.04 faster data processing and 4.48 lower evaluation perplexity than their counterparts, demonstrating superior computational efficiency and improved model quality. These properties stem from reduced communication overhead and the bounded growth of the global batch size relative to the federation group size.
This report is categorized under AITOP's "Papers" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include Llama3.1, federated learning, large model training.
Source attribution: curated from arXiv cs.AI (@Gianluca Mittone, Marco Aldinucci).
