这篇论文用1,600个人类写的幻觉样本对比了18,400个模型生成的样本，发现人类样本更稳定、更好控制，以后测幻觉不用老换模型了。
该研究构建了包含1,600个人类编写的幻觉样本数据集，覆盖中、英、法、意四种语言，并同时收集了来自五个视觉语言模型的18,400个样本进行对比。所有样本采用细粒度的跨度级标注方案来标记幻觉。实验发现，人类编写的样本在标注一致性上更高，且便于控制数据集内容，同时与模型生成样本在分布上保持相似。这表明人类数据可以替代模型生成的幻觉基准，用于更稳定的模型幻觉评估。
Can Humans Dream of Electric Sheep? Human-Written Samples for Fine-Grained Vision-and-Language Hallucination Benchmarking
In an age of rapid model turnover, how do we make hallucination evaluation more perennial? We explore whether human-written hallucination samples could take the place of model-generated hallucinations, in order to make benchmarking detection independent of particular models. To this end, we construct a dataset of 1,600 human-written samples, spanning four languages (Chinese, English, French, Italian), and 18,400 samples from five vision-and-language models, all annotated for hallucinations using a fine-grained span-level labeling scheme. We find that human-written samples result in higher agreement and allow greater control of dataset contents, while remaining distributionally similar to samples derived from vision-and-language samples and providing a reasonable portrayal of detection capabilities - suggesting that human data is a viable substitute for model-based hallucination benchmarks.