做新闻聚合或信息检索的开发者，这篇论文戳破了AI聊天机器人的“可靠”假象——高准确率不等于可信赖，自由回答和跨语言场景下漏洞明显，值得点开看看你的系统是否也踩了同样的坑。
一项新研究评估了商业AI聊天机器人在新闻问答中的表现，发现其在多选题上准确率超过90%，能回答几小时前的事件。然而，这种成功很脆弱：当需要自由回答、使用印地语或问题包含错误假设时，性能大幅下降。超过70%的错误源于检索失败或来源偏差，即系统找到了接近但不精确的证据，然后忠实但错误地回答。研究指出，这些模型并非因“不会思考”而失败，而是因定位到错误证据。这揭示了AI作为新闻中介的可靠性问题，提醒用户不要被高准确率迷惑。
AI chatbots can answer fresh news well, but their …
AI chatbots can answer fresh news well, but their weakest failures hide inside their confidence.
Best systems are surprisingly good at recent news when the question is clean and multiple choice.
But it also shows that this success is fragile, because the same systems get worse when they must answer freely, when the news is in Hindi, or when the user’s question contains a false assumption.
The best systems crossed 90% accuracy on multiple-choice questions about events reported only hours earlier, which means retrieval-augmented AI has moved from stale encyclopedia mode toward live information work.
That accuracy is not the same thing as reliability, because the systems were far worse when answers had to be produced freely
these models usually do not fail because they cannot “think,” but because they land on the wrong evidence.
More than 70% of errors came from retrieval failures or source divergence, where the system found something nearby but not exact, then answered faithfully from the wrong article, wrong language, wrong scope, or wrong timestamp.
----
Paper Link – arxiv. org/abs/2605.22785
Paper Title: "Evaluating Commercial AI Chatbots as News Intermediaries"