How to ensure software quality after AI agents write large amounts of code
For developers working with AI code assistants, this provides concrete, actionable strategies for managing quality and risk, including how to set up automated checks and review processes.
Based on first-hand experience from Claude Code core developers Boris Cherny and Addy Osmani, and operational insights from senior engineering leaders, the article outlines three key approaches. First, treat one-off code as black box with minimal impact, while production code must meet higher standards. Anthropic's internal safeguards include extensive linting, testing, Claude-driven end-to-end tests, daily fuzz testing, and automated code and security reviews. Second, use a phased upgrade path: start with the latest models (Opus 5 / Fable 5.1), increase effort levels, integrate Claude.md and Skills to teach Claude how to work in your codebase, and eventually have the model repay technical debt or rewrite the codebase. Third, align results with constraints by defining completion criteria and non-negotiable boundaries, then let the AI self-check by embedding exact build/test/lint commands in configuration and turning rejected issues into Skills. Review depth should be allocated based on blast radius, with higher standards for code involving money, authentication, or user data. Finally, instead of manually fixing errors, have the model document lessons learned in Claude.md or Skills.
AI Agent 大量写代码后软件工程质量怎么保障？ 来自 Claude Code 核心开发者 Boris Cherny 和 Addy Osmani 的一手经验 + 资深工程领导者的操作化提炼，非常值得学习。 # Boris Cherny 的核心贡献是“分层”和 Anthropic 的真实做法： 1. 一次性代码可以完全黑箱：反正要扔、出错影响小，不必追求完美。 2. 生产代码的标准要高于人写的。他在邮件中列出了 Anthropic 内部的实际护栏：大量 lint 规则、大量测试、Claude 驱动的端到端测试、每天运行的 Claude 驱动模糊测试、自动化代码评审和安全评审、自动化重构。 3. 他还给出了一条升级阶梯：换最新前沿模型（Opus 5 / Fable 5.1）→ 调高 effort（high/xhigh）→ 投入 CLAUDE.md 和 Skills 教会 Claude 在你的代码库工作 → 更强的人工引导 / 让 Claude 偿还技术债、重写代码库 → 等下一个模型。 # Addy Osmani 是把理念落成四个可执行动作： 1. 先对齐结果与约束：定义“完成”、划定不可触碰区域、判断复用是否优于新写，然后放手。他明确说最新模型不需要冗长的规划仪式——但要在坏改动变成 PR 前拒绝它。 2. 给 AI 自检手段：把确切的 build/test/lint 命令写进配置，把评审中反复拒绝的问题变成 Skills（/verify、e2e、schema 检查），开 PR 前运行。 3. 按 blast radius 分配审查深度：影响范围决定阅读投入；涉及 money、auth、用户数据的代码标准要高于人写的。 4. 出错不要悄悄手动修：让模型把教训写进 CLAUDE.md 或 Skills，这是最反直觉也最重要的一条。 # 把 Boris Cherny 和 Addy Osmani 两人观点合在一起看，有三个层次： 第一，质量保障的对象从“代码”变成了“系统”。 Boris 列出的那些护栏（lint、测试、fuzzer、自动评审）本质上都是自动化约束，而 Addy 点破：“质量现在存在于你围绕 agent 设置的约束中。”人不再审查产物本身，是设计让 AI 自我纠错的闭环。Anthropic 的做法证明这条路在工业规模上是可行的。 第二，审查资源按风险分配，而非平均分配。 “黑箱 vs 高标准”的分界线不是代码是谁写的，是 blast radius。这其实是对传统“所有代码都该被认真评审”理念的修正——AI 时代反而逼出了更理性的审查经济学。 第三，错误是资产而非麻烦。 Addy 第 4 条与 Boris 的升级阶梯形成互补：出错后第一反应不是人去修，是沉淀为 CLAUDE.md 规则或 Skills；若仍不行，再沿阶梯升级模型、调高 effort、偿还技术债。手动悄悄修复等于丢弃了最有价值的反馈信号。 Addy Osmani @addyosmani How do you hold the bar on production agent code?: 1. Agree on the outcome and the constraints first. What does "done" look like? what must it not touch? is the simpler design is to refactor or reuse what you already have? Then let Claude cook. You do not need a long planning ritual on the latest models. You do need to reject a bad change before it becomes a PR. 2. Give Claude a way to check its work. Put the exact build, test, and lint commands in there. Turn the things you reject in review into skills: /verify, e2e, schema checks and so on. Run those before you open the PR. Use /code-review. I've said that quality now lives in the constraints you put around your agents and think this is worth spending time on. 3. Your job is the design and the bar. Blast radius decides how much you read. Throwaway code with a small blast radius can be a black box. Production code should have a higher bar than if a human wrote it, especially anything that touches money, auth, or user data. 4. When Claude misses, don’t quietly fix it by hand. Have it write the lesson into CLAUDE.md or a skill. If it still misses, use the latest frontier model, turn effort to higher or have Claude pay down the debt and make the codebase easier to work in. You can start with one check you already run today on every PR. The rest compounds from there. 🔗 View Quoted Tweet 💬 1 🔄 0 ❤️ 1 👀 215 📊 1 ⚡
