Industry

Gary Marcus 引用基准数据称 4.2% 幻觉率拖累企业 ROI

Why we picked it

Gary Marcus 拿一个基准测试的 4.2% 幻觉率来解释为啥很多公司算不出 AI 的 ROI。如果你在给公司选型或评估 AI 落地成本,这条能帮你把错误率和人工复核开销挂上钩。写代码的朋友还能对照下自己的实际出错体验。

Gary Marcus 在 X 上引用一份基准测试结果,指出 AI 模型在该基准上的幻觉率为 4.2%,并认为这一错误率是企业难以从 AI 投资中获得回报的原因之一。他提醒该数字仅在特定基准下成立,不同场景结果会有差异。评论者 Skip Rhudy 回应称,自己在日常用 AI 写代码时的出错率略低于这一数字,但承认使用场景较窄。Marcus 强调,修补幻觉产生的错误本身需要额外的时间和资金成本,这直接侵蚀企业部署 AI 的收益。

Body

这条推文呼应了近期多份企业调研中普遍报告 AI 项目 ROI 不明的问题。4.2% 的错误率意味着每处理 100 项任务平均出现 4 次以上事实性错误,在需要人工复核的场景下,验证成本可能抵消自动化带来的效率提升。对开发者而言,在代码生成等错误可被编译器捕获的场景中影响较小,而在事实性要求高的业务场景中,幻觉率仍是选型时需要评估的核心指标。

Original · Gary Marcus

no wonder most companies aren’t reporting ROI. 4.2% hallucination rate (on this benchmark, mileage will vary) just ain’t good enough. fixing those errors costs time and money. Skip Rhudy @SkipRhudyWriter This is a bit worse than what I experience during daily work with AI tools, but my use case is narrow (code): @GaryMarcus tinyurl.com/75ae386h 🔗 View Quoted Tweet📊 3 ⚡

Industry Coordinates & Source Verification

This report is categorized under AITOP's "Industry" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows and US-China foundation model benchmarks and commercialization comparison. Key entities and technical topics under continuous tracking include GaryMarcus, hallucination rate, enterprise AI, AI ROI.

Source attribution: curated from Gary Marcus (@GaryMarcus).