CanvasAnneal: Curriculum RL for Diffusion Language Models
CanvasAnneal: Curriculum Reinforcement Learning for Diffusion Language Models
This research provides a new method for improving the reasoning capabilities of diffusion language models, which could lead to better performance on complex tasks.
This paper introduces CanvasAnneal, a curriculum-guided reinforcement learning framework for diffusion language models.
It addresses the exploration bottleneck in RL by injecting reasoning priors from a teacher model into the initial diffusion canvas.
The method improves performance on MATH500, Countdown, and Tau2 benchmarks compared to standard diffu-GRPO, accelerating reward improvement on several tasks.
CanvasAnneal: Curriculum Reinforcement Learning for Diffusion Language Models
Diffusion Language Models (DLMs) offer promising parallel generation capabilities but lag behind autoregressive models in complex reasoning and tool-use tasks. While Reinforcement Learning (RL) has recently been applied to enhance DLMs, standard RL approaches suffer from an exploration bottleneck. To address this, we inject reasoning priors from a stronger teacher model to guide RL exploration. In this paper, we introduce CanvasAnneal, a curriculum-guided diffusion RL framework. During the initial RL phase, we warm-start exploration by injecting teacher-generated reasoning traces into the initial diffusion canvas. As training progresses, we gradually remove this guidance and require the model to generate more of the reasoning trajectory independently. Across mathematical reasoning and tool-use benchmarks, CanvasAnneal improves over standard diffu-GRPO on MATH500, Countdown, and Tau2 and substantially accelerates reward improvement on several tasks, while gains are task-dependent. Our results suggest that structured training-time guidance can alleviate exploration bottlenecks in diffusion RL and speed up convergence on harder tasks.
This report is categorized under AITOP's "Papers" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include CanvasAnneal, diffusion language models, reasoning models, reinforcement learning.
Source attribution: curated from arXiv cs.LG (@Blake Olson, Yuhang Song, Emmett McQuinn, Yuan Shangguan).