想用LLM自动调超参？这个基准测了12个智能体，告诉你它们能做什么、卡在哪。
AgentHPOBench 提出一个包含30个可执行机器学习任务的序列化基准，覆盖7个研究类别，用于评估LLM智能体的超参数优化能力。每个任务从已验证的基线运行开始，智能体需逐步观察配置、指标和日志后提出下一组超参数。研究在统一协议下评测了12种智能体和传统HPO基线。结果显示当前智能体具备初步实验优化能力，但在持续迭代改进、复杂日志诊断和逼近参考性能方面仍有明显局限。
AgentHPOBench: A Benchmark For Evaluating LLM Agents as Sequential Hyperparameter Optimizers
As LLMs evolve from code completion systems into autonomous scientific agents, evaluating their ability to conduct experiments has become increasingly important. Existing benchmarks typically focus on static code generation, paper replication, or final answer correctness, but do not directly assess whether agents can interpret experimental evidence and use it to guide subsequent hyperparameter decisions. To address this gap, we introduce AgentHPOBench, a sequential benchmark comprising 30 executable machine learning tasks across seven research categories. Each task begins with a validated baseline run, after which an agent performs several sequential interventions. At each step, the agent observes the accumulated configurations, metrics, and logs before proposing the next valid configuration. We evaluate 12 widely used agents and conventional HPO baselines under a unified protocol. The results show that current agents exhibit measurable experimental optimization ability across domains, but still face clear limitations in sustained iterative refinement, complex log diagnosis, and consistent progress toward reported reference performance.