这篇论文用真实数据告诉你，同一个模型换种方式调用，成绩能差8个百分点，跑三次还不一样，评估AI安全时别只看一个数字。
一篇arXiv论文审计了ChatGPT聊天界面与OpenAI API两种访问模态在有无网页搜索下的表现，样本含401个提示、共4812次响应。结果显示，关闭搜索时聊天界面在两个基准上的准确率均低于API；启用搜索使准确率下降最多8个百分点，甚至逆转了一个基准的模态趋势。同一提示重复三次，最多21%的提示出现不一致响应。作者认为安全评估应系统纳入模态、多次运行一致性、搜索条件和引用行为等因素。
What Current AI Benchmarks Leave Unmeasured: Modality, Search, Citations, and Implications (for Safety Evaluations)
Large language model (LLM) benchmark evaluations are routinely used to support claims about model safety, reliability, and deployment readiness. Yet most evaluations rely on a single access modality (model APIs), perform a single run per prompt, and report accuracy as the primary outcome metric, without accounting for conditions such as web search that may have effects on model behavior in deployment. We audit these assumptions for one of the most widely-used LLMs, comparing two modalities, ChatGPT's chat UI and OpenAI's API, with and without web search enabled. We use a stratified total sample of 401 prompts from two popular benchmarks, BBQ and SafetyBench, collecting 4,812 total responses across three repeated runs per prompt. Beyond standard performance measures, we evaluate model output dimensions including response consistency, response text similarity, citation grounding, and abstention behavior. For instance, chat UI responses were less accurate than API responses on both benchmarks with search disabled. Enabling web search reduced accuracy by up to 8 percentage points, and even reversed the direction of modality performance trends for one benchmark. Repeated runs of the same prompt produced inconsistent responses in up to 21\% of prompts. The two modalities also grounded answers in different citations, and abstention behavior was also inconsistent across both modalities. These results illustrate that, even within a model family, reporting only simple accuracy metrics can obscure important forms of model behavioral variation relevant to AI safety assessments. We argue that AI safety evaluations should systematically account for modality, multi-run consistency, search conditions, and response-level behaviors to better reflect how deployed AI systems behave in practice.