Chinese contentHamel Husain 宣布一场以 AI 优化真实业务的竞赛
搞 AI 的朋友可以看看，这不是比谁的应用炫，而是比谁能用 AI 真正优化一门生意，还有评委阵容和送书活动。
#evals
19 stories · 2 in 7 days · 6 in 30 days
Stories, products and papers tagged "evals", newest first.
Nov 7
Nov 3
Chinese contentLegora CTO Jacob Lauritzen 谈为律师构建 AI 的实践细节
法律 AI 公司 Legora 的 CTO 聊了怎么做无 trace 的 evals 和百万格表格审查，还聊到按小时计费模式会被 AI 冲击，做垂直 AI 的可以看看。
Oct 20
Chinese contentAnthropic 讲解 Agent 权限设计：提议与执行分离
Anthropic 的人用 83 秒讲明白一个实用思路：让 Agent 只出方案不动手，执行前加人工审批，改 Agent 前先跑 evals。
Oct 19
Chinese contentThorsten Ball 给初级工程师的建议：用系统思维与智能体协作
Thorsten Ball 给初级工程师的回信，讲怎么用系统思维跟智能体协作，还提醒要早点搭 evals 来量产出。
Oct 17
Chinese contentAI Engineering 大会主旨：evals 是 AI 工程师不可替代的工作
aparnadhinak 在 AI Engineer NYC 大会上说：代码会越来越便宜，但帮你看数据、评估 evals 的活永远缺人，这活就是 AI 工程师的本职。
Oct 13
Lee Robinson's Stanford CS146S Lecture on Resident AI Agents
The flywheel structure — evals feeding daily harness fixes while retrained models close the quarterly loop — gives product teams a concrete operating rhythm most agent coverage skips. Engineering leads and indie builders planning agent products can use the 'build the minimum harness' heuristic to avoid UI work model progress will erase. The lecture also flags emergent multi-bot behaviour in group chats as a failure mode, citing the Moltbook and Hugging Face incidents as cautions.
Oct 8
Chinese contentNotion AI 负责人 Sarah Sachs：为什么你的产品需要自建评测
Notion 的 AI 负责人要讲怎么给自己的产品搭 evals，选模型别再只看跑分，做 agent 的可以关注这场 11 月 3 日的 Forge 活动。
Oct 6
Chinese contentHamel Husain：评估该多久跑一次？权衡成本与业务价值
Hamel Husain 回答了一个很实际的问题：evals 到底多久跑一次，他的答案是看成本、饱和度和业务价值，不是死记一个周期。
Sep 29
Sep 28
Chinese contentLogan Kilpatrick 预测 2027 年将出现规模化自动创收的智能体公司
Google 的 Logan Kilpatrick 放话 2027 年会有自动赚钱的智能体公司，还给了现在就能上手的自查方法，可以看看。
Chinese contentHamel Husain 谈 LLM 评测：LLM judge 本质是需要校验的分类器
Hamel Husain 回答了一个实用问题：用 LLM 当评审做评测时，得先拿人工标注数据验证它靠不靠谱，别直接信。
OpenAI updates image input handling in API, urges eval reruns
Model-side changes to image input handling can silently alter outputs in production pipelines that already passed earlier evals. Teams running vision workflows on the OpenAI API should diff their results against previous runs and read the changelog before shipping updates. Compared with locking model versions manually, this is a case where OpenAI itself flags behavioural drift.
Sep 27
Builders urged to learn engineering fundamentals before using agents
Agent outputs can look correct while hiding suboptimal or fragile solutions, so evaluation and engineering judgement stay essential even as coding assistants improve. Developers and technical leads adopting tools like Claude Code should invest in fundamentals — evals, architecture and concurrency — rather than delegating decisions entirely. The exchange offers a practical counterpoint to messaging that coding skills are no longer needed.
Sep 23
Chinese contentHamel Husain 与 Shreya Shankar 总结 AI 团队评估工作流的经验
Hamel Husain 和 Shreya Shankar 跟 50 多家 AI 公司合作后写的评估方法论，还附了个免费插件让编程 Agent 帮你搭评估，做产品的都该看看。
Evals Guide Sequel by Hamel Husain and Shreya Shankar
Concrete results show why eval investment pays: Ramp lifted receipt-collection accuracy from 35% to 83%, Harvey nearly doubled its contract reviewer quality score, and Cursor cut Auto Balance routing costs 41% while raising satisfaction. Teams building or scaling AI features can apply the guide's skipped-step framework and plugin directly. Practitioners hiring for AI product roles will also recognise why evals keep appearing in PM job requirements.
Hamel Husain and Shreya Shankar release advanced evals guide and skill
The guide identifies the specific step most teams skip when building eval pipelines and explains which parts of eval work should and should not be automated. PMs and engineers shipping AI features can install the free skill and let a coding agent generate evals instead of hand-writing them. With roughly half of 25 PM job listings now asking for evals experience, the skill doubles as a practical skill-building resource.
Hamel Husain and Shreya Shankar publish advanced AI evals guide
Eval skills are shifting from niche research work to a stated hiring requirement for product managers, which signals how AI teams now allocate engineering resources. Practitioners building or maintaining AI features can use the guide's automation advice and free plugin to set up eval loops without dedicated infrastructure. The cited cost and accuracy gains offer concrete baselines for teams benchmarking their own pipelines against peers.
Aug 20
Aug 9
