Primus Projection Cuts Distributed LLM Training Tuning to Single-Node Simulation
Beyond the Trial-and-Error Loop: Hybrid Projection and Automated Tuning for Distributed Training
Teams burning GPU-hours on OOM crashes and suboptimal parallelism recipes can instead score candidate configurations in seconds on one machine, then commit only to the promising ones. The paper's validated error rates at production scale — including on DeepSeek V3 and Qwen3 — make the projections credible for real deployments. Engineers planning training runs on 256 to 1,024 GPU clusters, or on hardware not yet available, get a practical tuning workflow they can adopt directly.
The paper introduces Primus Projection, a hybrid projection engine that combines single-device GPU microbenchmarks with a downscale-measure-upscale workflow to simulate multi-node LLM training configurations. Projections stay within roughly 10% of measured multi-node throughput, with 4.7% error on DeepSeek V3 671B at 1,024 GPUs and 1.9% error on Qwen3-235B-A22B at 256 GPUs. A companion Primus Tuning Agent uses the engine as a fast scoring oracle, sweeping axes deterministically and adding an LLM loop for cross-axis combinations. In a Mixtral 8x22B case study, it found a configuration with 27% higher throughput than a published 4-node BF16 reference, using under 30 minutes of single-node exploration. A CPU-only simulation mode also enables pre-silicon planning without any GPUs.
Beyond the Trial-and-Error Loop: Hybrid Projection and Automated Tuning for Distributed Training
Configuring distributed LLM training across hundreds of GPUs requires jointly optimizing tensor, pipeline, expert, context, and data parallelism alongside micro-batch size, recomputation, sharding, and overlap, a combinatorial space where memory legality can hinge on a single byte of activation accounting. Today, teams navigate it through expensive multi-node trial and error, burning GPU-hours on crashes and suboptimal recipes. We present Primus Projection, which closes this loop with two coupled components. First, a hybrid projection engine that measures what it can and simulates what it cannot: GPU microbenchmarks on as few as a single device capture compute kernels and any in-scope communication; a sub-node downscale-measure-upscale workflow analytically restores out-of-scope collectives, FSDP overlap, and the full pipeline schedule at the multi-node target. A pure simulation mode runs entirely on CPU, enabling pre-silicon planning without any GPU. Second, the Primus Tuning Agent treats the projection engine as a fast scoring oracle (seconds per candidate instead of tens of minutes) and uses it to drive an intelligent search over the configuration space. A deterministic seed planner sweeps single axes; an LLM investigation loop proposes cross-axis combinations the planner cannot reach, with an analytic memory pre-filter that rejects infeasible candidates before any GPU time is spent. Validated on Llama 3.1 (dense) and Mixtral 8x22B (MoE) across two GPU generations, all projections stay within ~10% of measured multi-node throughput, and fidelity holds at production scale: 4.7% error on DeepSeek V3 671B at 1,024 GPUs and 1.9% on Qwen3-235B-A22B at 256 GPUs. In a Mixtral 8x22B case study, the Primus Tuning Agent discovers a configuration delivering +27% throughput over the published 4-node BF16 reference at the same scale, in under 30 minutes of single-node exploration.
