Adaptive Testing Cuts Agent Evaluation Cost with 200 Questions
Continuous evaluation of production agents is usually too expensive to run often, and this paper shows a practical shortcut that keeps scores accurate to about one point. The transferable, difficulty-stratified subsets mean teams with any agent stack can adopt the method without recomputing item parameters each time. Builders running agent benchmarks on a budget can start with a one-day calibration window and scale from there.
A new paper (arXiv:2609.21267) shows how to re-evaluate a production analytics agent using only 200 questions, 38.
5% of its full benchmark, while reproducing the full score to within 1.03 points.
The authors analysed an agent serving tens of thousands of monthly users, drawing on 574 historical benchmark runs split into calibration and held-out date periods.
They compared random sampling, cached results, fixed representative subsets and adaptive testing based on item response theory, with multidimensional 2PL adaptive testing giving the best fidelity.
In production the team deployed difficulty-stratified fixed subsets instead, because they are simpler to run.Those subsets transferred to five other agent families without recalibration and stayed stable with calibration windows as short as one day.
Nice paper showing how to re-evaluate a production agent at a fraction of the cost. 200 questions, 38.5% of the full benchmark, reproduce the full score to within 1.03 points. The authors studied an analytics agent that serves tens of thousands of monthly users, using 574 historical benchmark runs split by date into calibration and held-out periods. They compared random sampling, cached results, fixed representative subsets and adaptive testing based on item response theory. Multidimensional 2PL adaptive testing gave the best fidelity. The team deployed difficulty-stratified fixed subsets instead because they are simpler to run. Those subsets transferred to five other agent families without recalibration and stayed stable with calibration windows as short as one day. Paper: arxiv.org/abs/2609.21267 Chat with Paper: academy.dair.ai/papers/efficie…📊 4 ⚡
This report is categorized under AITOP's "Papers" desk. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include item response theory, agents, evaluation, open-source models.
Source attribution: curated from elvis (@omarsar0).