Sebastian Raschka's Reasoning From Scratch Video 5 Covers Log-Probability Scoring
Log-probability scoring underlies loss functions like cross-entropy in pre-training and distillation, so the material transfers well beyond answer ranking. Practitioners building inference-time scaling or self-refinement pipelines can follow along with the timestamped PyTorch implementations and the MATH-500 evaluation setup. It is a hands-on complement to reading papers on reasoning methods.
Sebastian Raschka released round 5 of his 'Reasoning from scratch' video series.
The tutorial explains log-probability scoring and self-refinement, including how to compute token probabilities in PyTorch with attention to token indexing, shifted targets, and numerical stability.
Viewers build a rule-based scorer, implement a self-refinement loop that generates critiques and revised answers, and evaluate the approach on MATH-500.
Reasoning from scratch, round number 5! This time, talking about log-probability scoring (also a great fundamental concept for loss functions like cross-entropy in pre-training and distillation) and self-refinement.
00:00 Introduction and inference-time scaling recap 05:02 Loading the pretrained LLM 08:00 Comparing and scoring model answers 10:18 Building a rule-based scorer 17:53 Token probabilities and sequence likelihood 26:47 Computing token probabilities in PyTorch 30:12 Token indexing and shifted targets 37:27 Log probabilities and numerical stability 45:57 Scoring answers with average log probabilities 56:24 How self-refinement works 59:07 Generating critiques and revised answers 1:01:00 Implementing the self-refinement loop 1:05:57 MATH-500 evaluation results 1:07:35 Takeaways and next steps
This report is categorized under AITOP's "How-to" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include Sebastian Raschka, log-probability, self-refinement, MATH-500.
Source attribution: curated from Sebastian Raschka (@rasbt).
