PapersSelected

Amazon 与 UMass 论文:LLM critic 在 AutoML 搜索中收益有限

Why we picked it

Llama-3.1-8B 当搜索 critic 反而在 7/8 个数据集上拖后腿,最多丢 0.245 AUC,而理想 critic 也只能赚 0.005。做 agent 或 AutoML 的人该看看,加反思模块前先跑个无 critic 基线对比再说。

Amazon 与 University of Massachusetts 的论文测试了在 AutoML 搜索循环中加入 LLM critic 的效果,实验基于 96 个 pipeline 配置的固定搜索池。结果显示 Llama-3.1-8B 作为 critic 时,在 8 个数据集中的 7 个上表现低于无 critic 基线,最多损失 0.245 AUC。

Body

机制上,随机性的 critic 误判代价很小,因为循环会保留历史最优结果;但持续指向错误方向的建议会浪费搜索预算。理想 critic 的收益上限也只有约 0.005 AUC。当配置池扩大到 250 个、搜索轮次增加到 20 轮时,负面效应大部分消退。

论文结论是:在使用 LLM critic 之前,应先用相同预算跑一次无 critic 的基线,如果更差就直接去掉。这对在 agent 系统中默认堆叠 LLM 反思环节的团队是一个成本层面的提醒。

Original · rohanpaul_ai

New Amazon + University of Massachusetts paper finds that adding an LLM critic to a search loop has small upside, about 0.005 AUC at best, and much larger downside.

Weak LLM critic can push AutoML search below plain random search

Agent systems often add an LLM critic that reads past results and tells the search what to try next. Teams check whether it helps, rarely whether it hurts.

They ran the same tabular model search with and without a critic, over a fixed pool of 96 pipeline setups. Random critic mistakes cost almost nothing since the loop keeps its best result, but advice that steadily points the wrong way wastes tries.

A Llama-3.1-8B critic scored below the no-critic baseline on 7 of 8 datasets, losing up to 0.245 AUC. An ideal critic gained at most about 0.005 AUC, and the harm mostly faded with 250 setups and 20 rounds.

Before trusting an LLM critic, even a large one, run the same budget without it and drop it if it loses.

Industry Coordinates & Source Verification

This report is categorized under AITOP's "Papers" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include LLM critic, AutoML, Llama-3.1-8B, papers.

Source attribution: curated from rohanpaul_ai (@rohanpaul_ai).