Hamel Husain and Shreya Shankar publish advanced eval systems guide
源：https://t.co/zGoiQqJJ4c
Evals are shifting from a niche skill to a hiring requirement, with roughly half of the PM openings cited explicitly demanding eval experience. Teams building AI products can use the guide's automation boundaries and free coding-agent plugin to run eval work with fewer engineers. The company benchmarks give a concrete baseline for judging whether your own eval investment pays off.
Lenny Rachitsky announced an advanced sequel to Hamel Husain and Shreya Shankar's guide on building eval systems, drawing on their work with more than 50 AI companies. The guide covers the key step most teams skip, what to automate and what not to, and a free plugin that lets a coding agent handle much of the eval work. It cites concrete results: Ramp raised automatic receipt collection accuracy from 35% to 83%, Shopify shipped an AI workflow builder 2.2x faster and 68% cheaper than its frontier-model setup, Harvey nearly doubled its internal quality score on a rebuilt AI contract reviewer, and Cursor cut Auto Balance routing costs by 41% while raising user satisfaction. Nearly half of 25 PM job openings Rachitsky shared recently ask for eval experience.
源：https://t.co/zGoiQqJJ4c
源： x.com/lennysan/statu… Lenny Rachitsky @lennysan Evals have been coming up more and more in my conversations with podcast guests and PM friends. Nearly half of the 25 awesome PM job openings I shared last week ask for experience writing evals. And leading companies keep sharing what investing in evals bought them: — @tryramp took its automatic receipt collection from 35% to 83% accuracy. — @Shopify shipped an AI workflow builder that's 2.2x faster and 68% cheaper than the frontier-model setup it replaced. — @harvey__ai rebuilt its AI contract reviewer, nearly doubling its internal quality score. — @cursor_ai tuned its Auto Balance routing, with much higher user satisfaction at 41% lower cost. So I asked the 🐐s of evals, @HamelHusain and @sh_reya , to write an advanced sequel to their very popular "Building eval systems that improve your AI product." Drawing on their work with 50+ AI companies, they share the key step most teams skip, what you should (and shouldn't) automate, and a free plugin that lets a coding agent do most of the heavy lifting. Read it here: lennysnewsletter.com/p/advanced-eva… F 🔗 View Quoted Tweet 💬 0 🔄 0 ❤️ 0 👀 162 ⚡
