这篇论文用IRT给192个模型的安全基准做心理测量，能把评测成本砍掉97%以上，还能识破模型故意考差或API后门换模型。
一项研究将项目反应理论（IRT）应用于8个安全基准、192个语言模型，这是迄今最大规模的LLM安全评估心理测量分析。研究发现，拒绝严格度、真实性和情境危害三个可解释因子解释了模型间大部分差异。IRT选出的题目在恢复完整基准分数时误差低于随机子集，约10个自适应题即可替代若干单个基准，评估成本降低97%-99%。IRT还能用于检测模型的朴素沙袋行为和API后端模型更换。
Item Response Theory for AI Safety
Language models differ in how safely they behave and these differences are measured by safety benchmarks. But aggregated benchmark scores are hard to trust and interpret, because benchmarks duplicate one another, correlate heavily, and models may sandbag when they detect evaluation. To address these issues, we draw on Item Response Theory (IRT), a statistical toolkit for measuring these latents from performance on items with inferred psychometric properties. We fit IRT models to eight safety benchmarks across 192 language models, the largest psychometric analysis of LLM safety evaluations to date, and contribute three results. First, we find that three interpretable factors of refusal strictness, truthfulness, and contextual harm explain most of the variance between models across benchmarks. Second, psychometrically selected items recover full benchmark scores with lower error than random subsets of the same size, and roughly ten adaptively chosen items suffice for several individual benchmarks, cutting evaluation cost by 97-99%. Third, IRT supports audits of individual models, showing that it can be used to detect naive sandbagging and changes of model behind APIs. Overall, we show IRT is a ready-made toolkit for reading, reducing, and auditing safety benchmarks, which we recommend frontier labs and evaluators adopt.