Training Data and Routing Choices Push Open Models on ARC-AGI-2
Rules, Traces and Routing Statistics: Training and Calibration Data for Open Models on ARC-AGI-2
The clearest lesson is that most of the gain for the 27B model came from the serving-side inference procedure rather than fine-tuning data, which changes how practitioners should budget effort on reasoning benchmarks. Builders running small-model inference on constrained hardware can reuse the expert-pruning and KV-cache techniques directly. The study's 189-run methodology on the ARC Prize Foundation's semi-private set also offers a rare reproducible comparison for offline ARC-AGI-2 solvers.
A study tests which training and calibration data help three open Qwen models solve ARC-AGI-2 tasks offline on four L4 GPUs within 12 hours. For the NVARC Qwen3-4B grid model, a corpus of 12,000 executable rules and 95,993 tasks was built, but continued fine-tuning on 1,920 real task records alone lifted the semi-private score from 28.96 to 33.06 across five runs, with no synthetic-task checkpoint beating it. For Qwen3.8-27B, fine-tuning on 1,036 correct self-generated reasoning traces plus an inference procedure using a thinking cap and Python program checks raised the score from 26.81 to 44.03, near the base model's no-time-limit score of 42.36. Pruning Qwen3.8-Flash-Next (125B parameters, 6B active) from 512 to 296 experts per layer using ARC routing statistics enabled a 1.96M-token key-value cache and a best score of 56.67.
Rules, Traces and Routing Statistics: Training and Calibration Data for Open Models on ARC-AGI-2
We study which training and calibration data help three open models solve ARC-AGI-2 tasks when the solver runs offline on four L4 GPUs for at most 12 hours. We compare data offline on public tasks and report 189 runs on the 120 hidden semi-private tasks. For the NVARC Qwen3-4B grid model, we built a corpus of 12,000 executable rules in a typed language and 95,993 tasks; a rule is admitted only if its behaviour on a fixed probe set differs from that of every admitted rule, and a task only if its pairs refute a shortlist of nearest competing rules and defeat 12 shortcut solvers. Continued fine-tuning on 1,920 real task records with revised four-GPU scheduling raised the semi-private score from 28.96 (mean of the released model) to 33.06 (mean of five runs, best 34.86); no checkpoint trained with synthetic tasks, even all 95,993 of them, improved on the real-task model. For the Qwen3.8-27B reasoning model, we fine-tuned on 1,036 of its own reasoning traces with correct final answers and compared sets that add traces of other models and synthetic tasks. With an inference procedure built around the model (a thinking cap enforced in the serving engine, a Python program checked on the demonstrations after each answer, a long first attempt per input), the score rose from 26.81 to 44.03, the level the ARC Prize Foundation measured for the base model without a time limit (42.36); most of the gain came from the inference procedure. For the Qwen3.8-Flash-Next mixture-of-experts model (125B parameters, 6B active), GPU memory limited concurrency. Pruning from 512 to 384 experts per layer by router-weighted activation saliency grew the key-value cache from 0.55M to 1.39M tokens; a 296-expert list adapted with ARC routing statistics, keeping experts for program tokens, plus an FP8 indexer cache, reached 1.96M. With a tool that checks programs, the model scored 56.67, our best result.
