Hamel Husain and Shreya Shankar release advanced evals guide and skill
Pro tip: Install this new evals skill from @HamelHusain and @sh_reya, it'll save you many hours and ...
The guide identifies the specific step most teams skip when building eval pipelines and explains which parts of eval work should and should not be automated. PMs and engineers shipping AI features can install the free skill and let a coding agent generate evals instead of hand-writing them. With roughly half of 25 PM job listings now asking for evals experience, the skill doubles as a practical skill-building resource.
Lenny Rachitsky highlighted a new evals skill from Hamel Husain and Shreya Shankar, published as a sequel to their guide 'Building eval systems that improve your AI product'. The pair drew on work with more than 50 AI companies. The package includes a free plugin that lets a coding agent handle much of the eval setup. Cited results include Ramp lifting receipt collection accuracy from 35% to 83%, Shopify shipping a workflow builder 2.2x faster and 68% cheaper, Harvey nearly doubling its internal quality score, and Cursor cutting Auto Balance routing costs by 41%.
Pro tip: Install this new evals skill from @HamelHusain and @sh_reya, it'll save you many hours and ...
Pro tip: Install this new evals skill from @HamelHusain and @sh_reya , it'll save you many hours and a lot of mistakes github.com/ai-evals-cours… Lenny Rachitsky @lennysan Evals have been coming up more and more in my conversations with podcast guests and PM friends. Nearly half of the 25 awesome PM job openings I shared last week ask for experience writing evals. And leading companies keep sharing what investing in evals bought them: — @tryramp took its automatic receipt collection from 35% to 83% accuracy. — @Shopify shipped an AI workflow builder that's 2.2x faster and 68% cheaper than the frontier-model setup it replaced. — @harvey__ai rebuilt its AI contract reviewer, nearly doubling its internal quality score. — @cursor_ai tuned its Auto Balance routing, with much higher user satisfaction at 41% lower cost. So I asked the 🐐s of evals, @HamelHusain and @sh_reya , to write an advanced sequel to their very popular "Building eval systems that improve your AI product." Drawing on their work with 50+ AI companies, they share the key step most teams skip, what you should (and shouldn't) automate, and a free plugin that lets a coding agent do most of the heavy lifting. Read it here: lennysnewsletter.com/p/advanced-eva… F 🔗 View Quoted Tweet 💬 0 🔄 0 ❤️ 6 👀 1152 📊 2 ⚡
