Sebastian Raschka Releases Round 6 of Reasoning-from-Scratch Tutorial Series
The tutorial demystifies how reasoning models like DeepSeek-R1 are trained by letting viewers implement GRPO from scratch, including the KL term, verifiable rewards, and log-probability computation. Practitioners building or fine-tuning reasoning models can follow the code step by step and reproduce the MATH-500 evaluation on their own hardware. The GRPO-versus-PPO comparison is a useful mental model for anyone choosing an RL post-training method.
Sebastian Raschka published round 6 of his Reasoning from Scratch video series, a full implementation walkthrough of RLVR (Reinforcement Learning with Verifiable Rewards) and GRPO (Group Relative Policy Optimization).
The video covers the theory behind GRPO versus PPO, DeepSeek-R1's training approach, and a hands-on coding session from loading the pretrained model through computing the GRPO loss.
Training runs on MATH data with evaluation on MATH-500, including sections on memory requirements and training stability.
A full timestamped chapter list from introduction (00:00) to next steps (1:24:05) accompanies the release.
Reasoning from scratch, round number 6! An introduction (and implementation) of Reinforcement Learning with Verifiable Rewards (RLVR) and Group Relative Policy Optimization (GRPO).
00:00 Introduction 01:54 What makes a reasoning model different? 04:25 Reasoning traces and model capability 08:29 Accuracy and format rewards 11:34 Aha moments and DeepSeek-R1 training 14:41 Reasoning effort and answer length 18:38 RLHF and RLVR 23:04 GRPO vs. PPO 26:40 GRPO explained with a cooking analogy 31:43 The KL term and simplified GRPO 35:04 Loading the pretrained model 36:07 Loading the MATH training data 39:26 Sampling model responses 46:30 Computing verifiable rewards 49:55 Computing advantages 51:54 Token and sequence log probabilities 55:29 Implementing sequence log probabilities 57:37 Fixing the inference-mode error 1:02:24 Computing the GRPO loss 1:04:37 Putting the GRPO step together 1:09:19 The GRPO training loop 1:12:57 Training settings, logging, and checkpoints 1:17:24 Running training and inspecting outputs 1:19:28 Loading and evaluating checkpoints 1:22:33 MATH-500 results and training stability 1:24:05 Memory requirements and next steps
This report is categorized under AITOP's "How-to" desk. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include GRPO, RLVR, DeepSeek-R1, reinforcement learning.
Source attribution: curated from Sebastian Raschka (@rasbt).