Post-Training Delivery Benchmark Tests LLM Agents as Forward-Deployed Engineers
Trains but Doesn't Learn: A Post-Training Delivery Benchmark for LLM Agents as Forward-Deployed Engineers
The benchmark reframes agent evaluation around deliverable outcomes and payment-gated acceptance, a shift from metric-chasing benchmarks that miss corrupted training runs. Teams building or buying post-training-as-a-service can use the acceptance gate and corruption detector to screen automated fine-tuning pipelines. The human FDE baseline gives a concrete performance reference against all four frontier agents.
A paper on arXiv (2609.25237) introduces a benchmark measuring whether LLM agents can deliver fine-tuned models end to end, not just raise metrics. The agent drives ten delivery stages while an oracle scores each stage from platform-recorded facts, under budget, approval-gate and reproducibility constraints. The paper identifies a silent failure mode, TBDL, where loss falls and signals stay green but the delivered model matches the base model. Four agents were tested: Claude Opus 5, GPT-5.6-luna, Gemini 3.7 Flash and DeepSeek V4-Pro, on metered L40S, A100 and H200 GPUs across 8B to 70B open bases, with a human FDE arm as comparison.
Trains but Doesn't Learn: A Post-Training Delivery Benchmark for LLM Agents as Forward-Deployed Engineers
Post-training is becoming a service (PTaaS): a customer hands an operator data and a goal, and a forward-deployed engineer (FDE) returns a fine-tuned, evaluated, and deployed model under a budget, a human-approval gate, and reproducibility requirements. Seating an LLM agent in the FDE seat raises a question existing benchmarks cannot answer: not whether an agent can raise a metric, but whether it can be trusted to deliver. We answer it on a governed delivery plane, where an agent drives ten stages and an oracle scores each stage from platform-recorded facts. The central silent failure is the run that trains but does not learn (TBDL): loss falls, every signal stays green, and the delivered model is no better than the base. An operator-run acceptance gate catches every such run before payment, and a detector calibrated on known-corrupted runs flags severe corruption mid-run. We ran four frontier agents (Claude Opus 5, GPT-5.6-luna, Gemini 3.7 Flash, DeepSeek V4-Pro) end to end on metered L40S, A100, and H200 GPUs across 8B to 70B open bases, certifying every scenario before scoring. We also ran a human FDE arm under the same oracle and compare every agent against it.
