#LLM judge

3 stories · 1 in 7 days · 2 in 30 days
Stories, products and papers tagged "LLM judge", newest first.
Nov 10
PapersBuildersSelected00:32
UCLA and Northeastern paper finds 18% of agent tokens removable

The non-obvious takeaway is that token waste is a property of the task and harness, not the model, so harness design changes may cut costs more than switching to a cheaper model. Engineering teams running coding agents or long multi-step workflows can use the proxy-and-rerun method to audit their own traces. The Kimi-K3 result on SWE-bench Verified shows a 41% token cut with success intact, a concrete benchmark for anyone comparing agent efficiency across frontier models.

Event
rohanpaul_ai@rohanpaul_aiSource
Oct 27
How-toBuildersSelected12:23
Hamel Husain urges skepticism toward AI benchmark results

The practical point is that benchmark rankings generalize poorly to your own workloads, and a single trace review can expose grading or harness flaws in minutes. Builders choosing models for production should audit failing benchmark tasks against their own use cases and invest in private evals rather than trusting headline numbers. This applies whether you track US lab releases or open-source alternatives.

Hamel Husain@HamelHusainSource
Sep 28