FDE-Gym Benchmarks Agents as Forward-Deployed Engineers
The main failure mode is missing unstated constraints, which suggests conversational skill and requirement discovery matter more than coding ability for delivery-style agent work. Builders evaluating agent stacks can use the benchmark's penalty scheme to weigh interaction cost against task success. It offers a more realistic yardstick than static coding benchmarks like SWE-bench for teams deploying agents in client-facing settings.
FDE-Gym is a benchmark that evaluates agents as forward-deployed engineers on 95 real customer-delivery tasks. Agents must question scripted stakeholders, operate in production-like systems, and surface constraints the customer never stated. The best configuration across 13 setups from seven providers fully resolved only 14 of 95 tasks, with a top mean score of 0.56. Scoring is based on improvement to the customer's own success metric, with penalties for extra messages, trial runs and waiting time. Even on the 23 tasks where the best configuration gathered every key fact, it reached 0.8 or higher on only 10.
// FDE-Gym: Agents as Forward-Deployed Engineers //
They found that the best agent configuration fully resolved 14 of 95 real customer-delivery tasks.
FDE-Gym tests agents as forward-deployed engineers. Each task comes from a real delivery project. Agents must question scripted stakeholders, work in production-like systems, and find constraints the customer never stated.
A delivery is scored by how much it improves the customer's own success metric, with penalties for extra messages, trial runs, and waiting time.
Across 13 configurations from seven providers, the top mean score is 0.56. The main failure is missing unstated constraints. Even on 23 tasks where the best configuration collected every key fact, it reached 0.8 or higher on only 10.
Highly recommended benchmark to look at.
https://t.co/EwGp9jsGae
Chat with Paper: https://t.co/c8w3kBWyW6
