DeepSoftwareAnalytics团队推出PTA-IRT，能更高效评估软件工程代理，比传统方法更准确。
研究人员提出PTA-IRT框架，通过融合过程和结果信号来评估软件工程代理。该框架利用历史执行轨迹作为特权信息，包括探索的上下文、尝试的编辑和解决路径。在四个SWE基准测试中，PTA-IRT在低校准预算下始终优于先前的IRT基线。代码和数据已在GitHub公开。
Efficient SWE Agent Benchmarking via Trajectory-Aware Evaluation
Evaluating software engineering agents on realistic benchmarks is costly, since each task may require multi-step code exploration, modification, and test execution. Existing efficient evaluation methods select representative subsets to estimate full-benchmark performance, but are largely result-only: they fit historical pass/fail response matrices or static task semantics, discarding how agents solve problems. We propose PTA-IRT, a Privileged Trajectory-Aware Item Response Theory framework that fuses process and outcome signals. Historical execution trajectories supply process-level evidence beyond pass/fail, such as explored context, attempted edits, and solving paths, which PTA-IRT uses as privileged information for calibration subset selection and ability estimation. Under low calibration budgets, PTA-IRT consistently outperforms prior IRT baselines on score and ranking recovery across four SWE benchmarks. Code and data are publicly available at https://github.com/DeepSoftwareAnalytics/PTA-IRT.