Hamel Husain and Shreya Shankar publish advanced AI evals guide
Eval skills are shifting from niche research work to a stated hiring requirement for product managers, which signals how AI teams now allocate engineering resources. Practitioners building or maintaining AI features can use the guide's automation advice and free plugin to set up eval loops without dedicated infrastructure. The cited cost and accuracy gains offer concrete baselines for teams benchmarking their own pipelines against peers.
Lenny Rachitsky announced an advanced sequel to Hamel Husain and Shreya Shankar's guide "Building eval systems that improve your AI product", drawing on their work with 50+ AI companies. It covers the key step most teams skip, what to automate and what not to, and a free plugin that lets a coding agent handle much of the work. Rachitsky says nearly half of 25 PM job openings he shared ask for eval-writing experience. Cited results include Ramp raising automatic receipt collection accuracy from 35% to 83%, Shopify shipping a workflow builder 2.2x faster and 68% cheaper than its frontier-model setup, Harvey nearly doubling its contract reviewer's internal quality score, and Cursor cutting Auto Balance routing costs by 41%.
Evals have been coming up more and more in my conversations with podcast guests and PM friends. Nearly half of the 25 awesome PM job openings I shared last week ask for experience writing evals. And leading companies keep sharing what investing in evals bought them: — @tryramp took its automatic receipt collection from 35% to 83% accuracy. — @Shopify shipped an AI workflow builder that's 2.2x faster and 68% cheaper than the frontier-model setup it replaced. — @harvey__ai rebuilt its AI contract reviewer, nearly doubling its internal quality score. — @cursor_ai tuned its Auto Balance routing, with much higher user satisfaction at 41% lower cost. So I asked the 🐐s of evals, @HamelHusain and @sh_reya , to write an advanced sequel to their very popular "Building eval systems that improve your AI product." Drawing on their work with 50+ AI companies, they share the key step most teams skip, what you should (and shouldn't) automate, and a free plugin that lets a coding agent do most of the heavy lifting. Read it here: lennysnewsletter.com/p/advanced-eva… F 💬 23 🔄 12 ❤️ 148 👀 17456 📊 65 ⚡
