做 LLM 评测的团队终于有了自动化工具——Benchmark Agent 能持续生成新基准，避免模型性能饱和，建议做模型评估的开发者直接试试。
论文提出 Benchmark Agent，一个全自动构建 LLM/MLLM 评测基准的智能体系统。它从用户需求分析、子任务设计到数据标注和质量控制，全流程自动化。作者用它生成了 15 个覆盖文本理解、多模态理解和领域推理的基准，经人类评估和 LLM 评判验证，质量高且无需人工参与。该系统解决了传统基准构建劳动密集、易饱和的问题，能持续生成新基准以区分顶尖模型。代码和预览已公开。
Benchmark Everything Everywhere All at Once
Benchmarks are fundamental for evaluating and advancing LLMs and MLLMs by providing standardized and explicit measures of performance. However, their construction is labor-intensive and hard to reuse, raising concerns about sustainability and scalability. Moreover, existing benchmarks often quickly reach performance saturation after their release, resulting in insufficient discrimination among state-of-the-art models. To address these challenges, we introduce Benchmark Agent, a fully autonomous agentic system designed for benchmark building. Our framework orchestrates the complete benchmark construction pipeline, from user query analysis and subtask design to data annotation and quality control. To assess Benchmark Agent, we implement it to produce 15 representative benchmarks, spanning diverse evaluation scenarios, including text understanding, multimodal understanding, and domain-specific reasoning. Extensive experiments, including human evaluation, LLM-as-a-judge assessment, and consistency checks, demonstrate Benchmark Agent can generate high-quality benchmark samples with minimal human involvement. More importantly, through continual evaluation, we observe several insightful findings, including that current models struggle with certain domain-specific reasoning tasks. We believe that rapidly evolving benchmarks can contribute significantly to the research community. The preview and code will be publicly available at the demo page and code repository.