NVIDIA LoGRA Paper Cuts RL Training Memory Below LoRA Levels
The headline result is that full-parameter RL updates can fit in less memory than LoRA, removing a common excuse for freezing weights. Engineers blocked by GPU memory when running RL on larger models can adopt LoGRA and scale step sizes via its KL-budget control instead. Compared with rank-256 LoRA, it trades roughly two points of Pass @1 for nearly half the memory at 7B.
NVIDIA researchers published a paper (arXiv 2610.06647) on LoGRA, a memory-efficient method for post-training RL on LLMs. LoGRA compresses each weight gradient with a rank-256 random projection redrawn every step, avoiding full gradients for compressed matrices. At 1.5B parameters, LoGRA averages 7.2 GiB of training memory versus 13.2 GiB for rank-256 LoRA, with nearly equal Pass @4 (81.8% vs 81.4%). At 7B, memory per GPU drops 45.7% (31.8 to 17.3 GiB) while Pass @1 reaches 72.3% versus 72.5% for dense Adam. LoGRA also trains a 27B model for over 1,100 steps on a single eight-GPU node, where dense Adam runs out of memory.
Recommended NVIDIA paper on memory-efficient post-training RL. A very interesting technique for scaling RL for LLMs. The main takeaway is that you can update all of a model's weights with RL and still use less memory than LoRA. At 1.5B, LoGRA averages 7.2 GiB of training memory, while rank-256 LoRA needs 13.2 GiB. Pass @4 is about the same (81.8% and 81.4%), and Pass @1 is about 2 points lower. If GPU memory is what stops you from running RL on a bigger model, this method is worth a look. LoGRA never builds a full gradient for the weight matrices it compresses. As microbatches accumulate, each weight gradient is compressed with a rank-256 random projection that is redrawn every step, and the update is rebuilt from two small matrices. Before each step, it predicts how much the policy will change and scales the step to stay within a KL budget. Average training memory per GPU drops 45.7% at 7B (31.8 to 17.3 GiB), with peak accuracy close to dense Adam (72.3% and 72.5% Pass @1 ). It also trains a 27B model for more than 1,100 steps on a single eight-GPU node, where dense Adam runs out of memory. Paper: arxiv.org/abs/2610.06647 Chat with Paper: academy.dair.ai/papers/logra-s… 💬 3 🔄 2 ❤️ 11 👀 729 📊 6 ⚡
