New NanoGPT Speed Record at 39.9s via Flop-Aware Training
The record shifts NanoGPT training from kernel-level tuning to selectively skipping flops, meaning total parameters can grow far beyond active ones — up to 65B sparse embeddings on hardware as small as 8xH100. Practitioners reproducing speedrun records or designing sparse training recipes can study the PR and Deven's writeup for concrete techniques like retroactive beta2 and one-float optimizer states per 768 params. It is an open-source result on Keller Jordan's NanoGPT repo, directly comparable with prior speedrun baselines.
