AutoRLBench tests 10 frontier agents on failed RL runs
Debugging RL training is where agent evaluation meets real engineering pain, and a ~40% ceiling shows frontier models are far from automating it. Engineering teams running post-training pipelines can use the open code and configs to benchmark which agents catch data-level faults before wasting GPU hours. Compared with synthetic bug suites, its use of authentic failed runs makes the scores harder to game.
