这篇论文用252个真实bug数据，告诉你怎么测试全绿依然出bug，给所有做LLM应用的人敲响警钟。
该论文分析了一个生产级租赁搜索助手，其自动化测试套件在六周内增长到1,553个测试用例。尽管套件持续通过，用户可见缺陷仍不断进入生产环境。研究检查了项目中所有252个bug修复提交，发现约44%的修复属于四个无法被组件级单元测试观察的接缝：实时浏览器运行时、非默认市场、端到端流程和全系统级别。一个缺陷因缺少接缝防护而两次发布。论文提出了四接缝框架和实测缺陷分布。
All Green, Still Broken: Real-Flow Verification Lessons from an LLM-Integrated, Multi-Market Web Application
Modern web applications increasingly combine three ingredients that are hard to test: output from large language models, multi-market internationalization, and browser-driven front-ends over external data sources. We report on a production rental-search assistant whose automated suite grew to 1,553 test cases in six weeks. The suite passed continuously, yet user-facing defects continued to reach production. We studied all 252 bug-fix commits in the project and classified each by the boundary, or seam, it escaped through. About 44 percent of the fixes fall in four seams that component-level unit tests cannot observe: the live browser runtime, the non-default market, the end-to-end flow, and the whole-system level. A fix without a guard at the seam let one defect ship twice. We present the four-seam framework, the measured defect distribution, and the practices we adopted, including a simple way for a team to find the seam that carries the most fixes.