TensorTonic explains benchmark evaluation from first principles
The practical shift is treating evaluation as a design problem: runtime and grader choices can swing scores by tens of points, so leaderboard rankings say more about harnesses than models. Teams choosing models for production should track error rates and abstention behaviour alongside accuracy, and re-run benchmarks under their own scaffolding. Engineers building agent workflows or buying based on vendor benchmarks will find the pass@k versus pass^k framing directly applicable to release gates.