The non-obvious takeaway is that token waste is a property of the task and harness, not the model, so harness design changes may cut costs more than switching to a cheaper model. Engineering teams running coding agents or long multi-step workflows can use the proxy-and-rerun method to audit their own traces. The Kimi-K3 result on SWE-bench Verified shows a 41% token cut with success intact, a concrete benchmark for anyone comparing agent efficiency across frontier models.
#LLM judge
3 stories · 1 in 7 days · 2 in 30 days
Stories, products and papers tagged "LLM judge", newest first.
Nov 10
UCLA and Northeastern paper finds 18% of agent tokens removable
Oct 27
Hamel Husain urges skepticism toward AI benchmark results
The practical point is that benchmark rankings generalize poorly to your own workloads, and a single trace review can expose grading or harness flaws in minutes. Builders choosing models for production should audit failing benchmark tasks against their own use cases and invest in private evals rather than trusting headline numbers. This applies whether you track US lab releases or open-source alternatives.
Sep 28
Chinese contentHamel Husain 谈 LLM 评测:LLM judge 本质是需要校验的分类器
Hamel Husain 回答了一个实用问题:用 LLM 当评审做评测时,得先拿人工标注数据验证它靠不靠谱,别直接信。