Unified Per-Token Gating Family for On-Policy Distillation
This paper presents a new parameterization for per-token gating in on-policy knowledge distillation, which is a key technique for training smaller models to mimic larger ones. It provides a framework for comparing different gating designs.