swyx argues independent AA-* rebuilds are needed to catch benchmaxxing
The core implication is that official benchmark scores are increasingly unreliable signals for model selection, and community-run alternative evals are becoming the de facto quality check. Builders choosing models for production should cross-check lab claims against independent altbenches before committing to an API or fine-tuning pipeline. Following Artificial Analysis's rebuilds offers a practical sniff test without waiting for lab corrections.
swyx announced on X that he is recording an AAv5 podcast episode with @_micah_h and @grmcameron. In the post he argued that labs cannot be trusted to self-report benchmark results, since they may intentionally or accidentally benchmaxx. He called for trusted third parties like Artificial Analysis to rebuild every major eval under the AA-* banner. His proposed method ranges from simple perturbations like randomizing multiple-choice answer order to generating new problems that test the same underlying reasoning steps.
we're recording an AAv5 pod with @_micah_h and @grmcameron today and this reminds me why you basically need @ArtificialAnlys AA-* rebuilds of every single eval
because you -CANNOT- trust labs not to intentionally or accidentally benchmaxx, you need a few trusted third parties to basically redo the entire eval in spirit but not in letter, to have a quick sniff test that model claims hold up under minor perturbations (eg starting from "randomizing order in multiple choice questions" to coming up with completely new problems that superficially look different but actually test the same underlying logic and reasoning steps). not enough people are reading bench traces like this 👇 and the systematic way to catch this benchmaxxing is basically to have altbenches for everything
