PKPS 方法利用稀疏缓存响应预测生成模型基准成绩
Benchmark Prediction with Sparse and Unpaired Cached Responses
这篇论文提出 PKPS，只跑部分查询就能预测模型在整套基准上的分数，能省下大量评估算力，做模型评测的可以看看。
评估生成模型时，通常只跑基准任务的部分查询，导致分数矩阵稀疏且不成对。论文提出 Product Kernel Perspective Space（PKPS），基于模型的嵌入响应构建低维表示，通过比较相似查询而非仅相同查询来利用稀疏、不成对的数据。理论上，PKPS 相比只使用稠密成对数据的方法在查询效率上更优。实验显示 PKPS 在两个大型基准套件上的基准预测超过了基线方法。
Benchmark Prediction with Sparse and Unpaired Cached Responses
Evaluating a generative model on all queries from all available benchmark tasks is prohibitively expensive. As such, models are often evaluated on different subsets of tasks and/or different subsets of queries from a given task. The resulting collection of responses and the corresponding (model, task) score matrix are sparse and unpaired. In this paper, we introduce the Product Kernel Perspective Space (PKPS), a collection of low-dimensional representations of models based on their embedded responses. PKPS generalizes existing methods for representing black-box generative models by comparing responses to similar -- rather than just identical -- queries and is therefore able to utilize sparse and unpaired information. Theoretically, we show that benchmark prediction using PKPS is query-efficient relative to methods that only use information from dense and paired portions of the evaluation data. Empirically, we show that PKPS-based methods outperform baselines on benchmark prediction across two large benchmark suites.
