Amazon and UMass paper finds LLM critics can hurt AutoML search
The asymmetry is the point: even a large critic model can systematically mislead search, while the theoretical ceiling on benefit is only about 0.005 AUC — so the default of bolting an LLM onto a search loop deserves scrutiny. ML engineers and agent builders should treat this as a cheap ablation checklist item: benchmark any critic against a critic-free run at equal budget before shipping it. The finding also applies to agentic pipelines beyond AutoML where LLM suggestions steer exploration.