这篇论文把EffiBench、EvalPerf等流行代码性能基准的漏洞全翻出来了——1538个任务里只有6%能真正测出差异。他们搞了个多智能体框架自动生成更刁钻的测试，效果直接翻四倍。
这篇论文重新评估了EffiBench、Enamel、EvalPerf和Mercury四个代码性能基准，对1,538个任务进行30次运行统计检验。结果显示，基准提供的优化实现中仅有6.11%明显快于规范解法。在308个非显著的任务中，有209个存在潜在的性能优化但未被原始测试暴露。为此，作者提出一个LLM多智能体框架（使用DeepSeek-v3.1和GPT-4o）生成性能导向测试，在1,345个原始测试无显著差异的任务中，分别有24.01%和25.43%检测出显著差异。
Rethinking Code Performance Benchmarks for LLMs
Many function-level performance benchmarks have been proposed to evaluate whether large language models (LLMs) can generate efficient programs. However, results on these benchmarks often show that LLM-generated implementations have little or no execution-time difference from canonical solutions. In this paper, we revisit four popular benchmarks: EffiBench, Enamel, EvalPerf, and Mercury. We evaluate 1,538 tasks under more rigorous setting by running each task 30 times and assessing the runtime differences between the canonical solutions and benchmark-provided performant implementations with statistical testing. With the benchmark-provided test suites, only 6.11% of the performant implementations are significantly faster than the canonical solutions. In a manual analysis of 308 non-significant tasks, 99 performant implementations contain no meaningful performance change, while 209 contain potential performance improvements that are not exposed by the original tests. These results suggest that the main limitation is not only the evaluation method, but also the limited sufficiency of the benchmark-provided performance tests. To address this limitation, we propose an LLM-based multi-agent framework to generate performance-oriented tests that expose runtime differences more effectively than the original tests. The framework uses three separate agents to generate, diagnose, and repair deterministic tests that preserve functional correctness while better exposing performance differences. Across 1,345 benchmark tasks for which the original tests found no significant performance difference, tests generated by our framework with DeepSeek-v3.1 and GPT-4o reveal statistically significant improvements in 24.01% and 25.43% of the tasks, respectively, outperforming the SOTA LLM-based performance test generation method.