PTA-IRT Framework for SWE Agents
Efficient SWE Agent Benchmarking via Trajectory-Aware Evaluation
PTA-IRT reduces evaluation costs for AI coding assistants while maintaining accuracy, making it valuable for developers and researchers in ASEAN.
PTA-IRT improves software engineering agent evaluation by using historical execution trajectories as privileged information. The framework outperforms prior IRT baselines on score and ranking recovery across four SWE benchmarks. It explores context, attempted edits, and solving paths beyond simple pass/fail results. Code and data are publicly available at the GitHub repository.
Efficient SWE Agent Benchmarking via Trajectory-Aware Evaluation
Evaluating software engineering agents on realistic benchmarks is costly, since each task may require multi-step code exploration, modification, and test execution. Existing efficient evaluation methods select representative subsets to estimate full-benchmark performance, but are largely result-only: they fit historical pass/fail response matrices or static task semantics, discarding how agents solve problems. We propose PTA-IRT, a Privileged Trajectory-Aware Item Response Theory framework that fuses process and outcome signals. Historical execution trajectories supply process-level evidence beyond pass/fail, such as explored context, attempted edits, and solving paths, which PTA-IRT uses as privileged information for calibration subset selection and ability estimation. Under low calibration budgets, PTA-IRT consistently outperforms prior IRT baselines on score and ranking recovery across four SWE benchmarks. Code and data are publicly available at https://github.com/DeepSoftwareAnalytics/PTA-IRT.
