PapersSelected

Meta paper reports ML engineering agents running 1,097 search experiments

Why we picked it

The non-obvious takeaway is that harness design and objective calibration, not model strength, drove the jump in acceptance rate from 15.6% to 29.6%. Practitioners building agent-driven experiment loops can copy the three mitigations directly: parallel runs, online-calibrated objectives, and disciplined context engineering. Engineering leaders weighing agent adoption in ML pipelines get a rare production-scale, A/B-tested reference point instead of a demo.

Meta published a paper on ML engineering agents validated through a large-scale A/B test on production search rankers. In one campaign, the system completed 1,097 experiments and accepted 325 of them, a 29.6% acceptance rate, up from 15.6% in an earlier campaign with a different objective. The best accepted model raised search volume by 0.23% in a live test.

Body

The agents propose a model change, implement it, train it, and evaluate it, while engineers retain control of the objective, the search direction, and final review. The team identified three core difficulties: each experiment takes hours, offline metrics often disagree with online impact, and gains flatten after early wins. Their answers were parallel training runs, an objective calibrated to online impact, and careful context engineering. Notably, several accepted changes were architectures the team had never tried before.

The paper's broader lesson is that a stronger model alone was insufficient; the surrounding loop mattered just as much. Teams deploying agents on training pipelines are advised to calibrate the objective to online impact first.

Original · DAIR.AI

Great paper from Meta on ML engineering agents, tested with a large-scale A/B test on production search rankers.

(bookmark it)

They also show how important the harness design is.

The agents propose a model change, implement it, train it, and evaluate it. Engineers keep control of the objective, the search direction, and the final review.

In one campaign, the system completed 1,097 experiments and accepted 325 of them. That is a 29.6% acceptance rate, against 15.6% in an earlier campaign with a different objective. The best model raised search volume by 0.23% in the live test.

They share that three problems made this hard. Each experiment takes hours, offline metrics often disagree with online impact, and gains flatten after the early wins. The team answered with parallel training runs, an objective calibrated to online impact, and careful context engineering.

Several accepted changes were architectures the team had never tried.

The authors also report that a stronger model was not enough on its own. The loop around it mattered just as much. If you plan to put agents on your training pipeline, calibrate the objective first.

https://t.co/OeZdpd9xTQ

Chat with Paper: https://t.co/DpjKciVwe6

Industry Coordinates & Source Verification

This report is categorized under AITOP's "Papers" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include Meta, agents, papers.

Source attribution: curated from DAIR.AI (@dair_ai).