Meta's Benchmark Factory Turns Internal Code Changes into Coding Evals
The non-obvious takeaway is that per-team evals can now be generated from a repo's own merged diffs, so model selection can reflect code that actually ships rather than public benchmark scores. Engineering leads choosing models for coding assistants, and agent builders needing regression tests, can run the loop against their own repos. It offers a concrete contrast with fixed public suites like SWE-bench, which rarely match internal codebases.